Objective To examine how β-amyloid (Aβ), APOE and BDNF genotypes, and cortisol relate to depressive and anxiety symptoms in cognitively normal older women and men. Methods Cross-sectional data were analyzed from 423 older adults from the Australian Imaging Biomarkers and Lifestyle study. Analyses of covariance evaluated associations between Aβ, APOE and BDNF genotype, and cortisol in relation to severity of depressive and anxiety symptoms. Results Among Aβ+ older adults, APOE ε4 carriage was associated with greater severity of anxiety symptoms (d = 0.55); and in the full sample, APOE ε4 carriage was linked to greater severity of depressive (d = 0.26) and anxiety (d = 0.21) symptoms. Among Aβ+ women, ε4 carriers reported greater anxiety symptoms than non-ε4 carriers (d = 0.83), and female BDNF rs6265 Val66 Met allele carriers reported greater depressive symptoms (d = 0.29). Conclusion Sex moderated the relationship between Aβ, APOE genotype, and BDNF genotype in predicting severity of anxiety and depressive symptoms in cognitively normal older adults.