Background: We aimed to determine the prospective association between measured hearing impairment, self-reported hearing handicap and hearing aid use with quality of life. Study design: 829 Blue Mountains Hearing Study participants (≥55 years) were examined between 1997-1999 and 2007-2009. The shortened version of the hearing handicap inventory was administered. Hearing levels were measured using pure-tone audiometry. Quality of life was assessed using the 36-Item Short-Form Survey (SF-36); higher scores reflect better quality of life. Results: Hearing impairment at baseline compared with no impairment was associated with lower mean SF-36 mental composite score 10 years later (multivariable-adjusted p = 0.03). Physical composite score and mean scores for seven of the eight SF-36 domains after 10-year follow-up were significantly lower among participants who self-reported hearing handicap at baseline. Differences in the adjusted means between participants with and without hearing handicap ranged from 2.7 (physical composite score) to 10.4 units ('role limitations due to physical problems' domain). Individuals who developed incident hearing impairment compared to those who did not, had adjusted mean scores 9.5- and 7.7-units lower in the 'role limitation due to physical problems', and 'bodily pain' domains, respectively, at the 10-year follow-up. Hearing aid users versus non-users at baseline showed a 1.82-point (p = 0.03) and 3.32-point (p = 0.01) increase in SF-36 mental composite score and mental health domain over the 10-year follow-up, respectively. Conclusion: Older adults with self-perceived hearing handicap constitute a potential risk group for overall deterioration in quality of life, while hearing aid use could help improve the well-being of hearing impaired adults.