Objective: Assess Aβ deposition longitudinally and explore its relationship with cognition and disease progression. Methods Clinical follow-up was obtained 20 ± 3 months after [11C]Pittsburgh compound B (PiB)-positron emission tomography in 206 subjects: 35 with dementia of the Alzheimer type (DAT), 65 with mild cognitive impairment (MCI), and 106 age-matched healthy controls (HCs). A second PiB scan was obtained at follow-up in 185 subjects and a third scan after 3 years in 57. Results At baseline, 97% of DAT, 69% of MCI, and 31% of HC subjects showed high PiB retention. At 20-month follow-up, small but significant increases in PiB standardized uptake value ratios were observed in the DAT and MCI groups, and in HCs with high PiB retention at baseline (5.7%, 2.1%, and 1.5%, respectively). Increases were associated with the number of apolipoprotein E ε4 alleles. There was a weak correlation between PiB increases and decline in cognition when all groups were combined. Progression to DAT occurred in 67% of MCI with high PiB versus 5% of those with low PiB, but 20% of the low PiB MCI subjects progressed to other dementias. Of the high PiB HCs, 16% developed MCI or DAT by 20 months and 25% by 3 years. One low PiB HC developed MCI. Interpretation Aβ deposition increases slowly from cognitive normality to moderate severity DAT. Extensive Aβ deposition precedes cognitive impairment, and is associated with ApoE genotype and a higher risk of cognitive decline in HCs and progression from MCI to DAT over 1 to 2 years. However, cognitive decline is only weakly related to change in Aβ burden, suggesting that downstream factors have a more direct effect on symptom progression.