It is unclear what the contribution of prenatal versus childhood development is for adult cognitive and sensory function and age-related decline in function. We examined hearing, vision and cognitive function in adulthood according to self-reported birth weight (an index of prenatal development) and adult height (an index of early childhood development). Subsets (N = 37,505 to 433,390) of the UK Biobank resource were analysed according to visual and hearing acuity, reaction time and fluid IQ. Sensory and cognitive performance was reassessed after ∼4 years (N = 2,438 to 17,659). In statistical modelling including age, sex, socioeconomic status, educational level, smoking, maternal smoking and comorbid disease, adult height was positively associated with sensory and cognitive function (partial correlations; pr 0.05 to 0.12, p < 0.001). Within the normal range of birth weight (10th to 90th percentile), there was a positive association between birth weight and sensory and cognitive function (pr 0.06 to 0.14, p < 0.001). Neither adult height nor birth weight was associated with change in sensory or cognitive function. These results suggest that adverse prenatal and childhood experiences are a risk for poorer sensory and cognitive function and earlier development of sensory and cognitive impairment in adulthood. This finding could have significant implications for preventing sensory and cognitive impairment in older age.