Unlike other life domains, sexual quality of life (SQoL) has a negative relationship with age. This study disentangled the effect of age in this relationship from confounding sociocultural influences (e.g., the period of time in which data were collected, and cohort differences) and aimed to understand the roles of other sexual domains (i.e., frequency, perceived control, thought and effort invested in sex, and number of sexual partners). We analyzed data from the longitudinal Midlife in the United States study (n = 6,278; age range 20–93), which were collected between 1995 and 2013. Repeated measures linear mixed-effects models showed that age was the most robust time-related predictor of declining SQoL. However, after the sexual domains were included in the model, age had a positive relationship with SQoL and older adults’ SQoL was differentially influenced by the quality—not quantity—of sex. When partnership characteristics were included in the model, age was no longer related to SQoL. These findings suggest that aging may be associated with the acquisition of skills and strategies that can buffer age-related declines in SQoL, particularly in the context of a positive relationship. We summarize these findings as sexual wisdom.