Objective:To evaluate the combined effect of lesion activity and pathologic processes occurring in both chronically demyelinated lesions and normal-appearing white matter (NAWM) on brain atrophy in MS. Methods: Pre- and post-gadolinium T1, fluid attenuation inversion recovery, and diffusion tensor imaging images were acquired from 50 consecutive patients with relapsing-remitting MS (all, but one, on disease-modifying therapy) at baseline and 5 years. Brain atrophy was measured using structural image evaluation, using normalization of atrophy percent brain volume change (PBVC) analysis. Results: During follow-up, brain volume diminished by 2.0% ± 1.1%. PBVC was not associated with patient age, disease duration, sex, or type of treatment. PBVC moderately correlated with baseline lesion load (r = -0.38, p = 0.016), but demonstrated strong association with new lesion activity (r = -0.63, p < 0.001). Brain atrophy was also strongly linked to the increase of water diffusion within chronic MS lesions (r = -0.62, p < 0.001). In normal-appearing white matter (NAWM), PBVC demonstrated a significant correlation with both baseline and longitudinal increase of demyelination as measured by radial diffusivity (RD, r = -0.44, p = 0.005 and r = -0.35, p = 0.026, respectively). Linear regression analysis explained 62% of the variance in PBVC. It confirmed the major role of new lesion activity (p = 0.002, standardized beta-coefficient -0.42), whereas change in diffusivity inside chronic lesions and NAWM RD at baseline also contributed significantly (p = 0.04 and 0.02, standardized beta-coefficient -0.31 and -0.29, respectively), increasing predictive power of the model by 55%. Conclusion: In addition to new lesion activity, progressive loss of demyelinated axons in chronic lesions and the degree of demyelination in NAWM significantly contribute to accelerated loss of brain tissue in patients with MS receiving immunomodulatory therapy.