The near-infrared is an important part of the spectrum in astronomy, especially in cosmology because the light from objects in the early universe is redshifted to these wavelengths. However, deep near-infrared observations are extremely difficult to make from ground-based telescopes due to the bright background from the atmosphere. Nearly all of this background comes from the bright and narrow emission lines of atmospheric hydroxyl (OH) molecules. The atmospheric background cannot be easily removed from data because the brightness fluctuates unpredictably on short timescales. The sensitivity of ground-based optical astronomy far exceeds that of near-infrared astronomy because of this long-standing problem. GNOSIS is a prototype astrophotonic instrument that utilizes "OH suppression fibers" consisting of fiber Bragg gratings and photonic lanterns to suppress the 103 brightest atmospheric emission doublets between 1.47 and 1.7 μm. GNOSIS was commissioned at the 3.9 m Anglo-Australian Telescope with the IRIS2 spectrograph to demonstrate the potential of OH suppression fibers, but may be potentially used with any telescope and spectrograph combination. Unlike previous atmospheric suppression techniques GNOSIS suppresses the lines before dispersion and in a manner that depends purely on wavelength. We present the instrument design and report the results of laboratory and on-sky tests from commissioning. While these tests demonstrated high throughput (≈60%) and excellent suppression of the skylines by the OH suppression fibers, surprisingly GNOSIS produced no significant reduction in the interline background and the sensitivity of GNOSIS+IRIS2 is about the same as IRIS2. It is unclear whether the lack of reduction in the interline background is due to physical sources or systematic errors as the observations are detector noise dominated. OH suppression fibers could potentially impact ground-based astronomy at the level of adaptive optics or greater. However, until a clear reduction in the interline background and the corresponding increasing in sensitivity is demonstrated optimized OH suppression fibers paired with a fiber-fed spectrograph will at least provide a real benefit at low resolving powers.