Sun exposure is the main cause of squamous (SCC) and basal cell carcinoma (BCC) although pattern and amount differ by cancer type, and sun sensitivity is the major host risk factor. Our study investigated risk factors and residential ambient UV in a population-based sample of Australian 45 and Up Study participants: 916 BCC cases, 433 SCC cases, 1224 controls. Unconditional logistic regression models adjusting for key covariates demonstrated 60% increased BCC risk and two-fold increased SCC risk with sun sensitivity, and three- and four-fold increased risk, respectively, with solar keratoses. BCC but not SCC risk increased with higher early-life residential UV in all participants (odds ratio (OR) = 1.54; 95% CI 1.22–1.96 for intermediate; OR = 1.31; 95% CI 1.03–1.68 for high UV at birthplace) and similarly in Australian-born participants (P-values < 0.05). Risk of SCC but not BCC increased with long-term cumulative sun exposure assessed by self-reported outdoor work (OR 1.74, 95% CI 1.21–2.49). In conclusion, sun sensitivity is important for both cancers, early-life UV but not cumulative UV appears to increase BCC risk, the former an apparently novel finding, and SCC risk appears only to be related to long-term cumulative sun exposure.