PURPOSE. To compare white matter (WM) injuries associated with vision loss in multiple sclerosis (MS) and optic neuritis (ON). METHODS. Twenty-three patients with clinically definite relapsing-remitting MS and chronic unilateral ON and 14 neurologically healthy volunteers were monocularly tested with Sloan 100%, 2.5%, and 1.25% contrast visual acuity charts. Primary visual pathway and whole-brain WM injury were assessed with optical coherence tomography (OCT) and diffusion tensor imaging (DTI). OCT and DTI correlates of high- and low-contrast visual impairment were identified using correlation analyses. RESULTS. The MS patients displayed significantly reduced retinal nerve fiber layer (RNFL) thickness and altered optic nerve and radiation DTI measures compared with the controls. In the patients, 2.5% and 1.25% contrast letter acuity in the unaffected eye correlated significantly and independently with optic nerve and optic radiation DTI measures. Visual acuity in affected eyes did not correlate with optic nerve or optic radiation DTI measures, but did correlate with DTI measures in prefrontal and temporal brain regions that were shown to connect structurally to visual cortices. CONCLUSIONS. In unaffected eyes, visual impairment was associated with WM injury in the visual pathway. In contrast, irrecoverable visual impairment after ON was associated with injury to frontal WM, which potentially impairs the capacity for remapping visual processing.