Background: To explore the association between sexual activity and change in cognitive function over 4 years in a representative sample of older adults in England. Methods: Data were from 1963 men and 2513 women participating in Wave 6 (2012/2013) and Wave 8 (2016/2017) of the English Longitudinal Study of Ageing. Participants reported whether or not they had engaged in any sexual activity in the last year. Cognitive function was assessed with tests of immediate and delayed recall. Adjusted general linear models were used to test associations between sexual activity and changes in cognitive function. Results: Men who were sexually active at baseline had better preservation in immediate (0.18 points, 95% CI 0.07–0.29, p = 0.002) and delayed recall (0.19 points, 95% CI 0.08–0.29, p = 0.001) over 4-year follow-up. No significant associations were observed for women. Discussion: Strengths of this study include large, representative sample, longitudinal design and adjustment for a wide range of potential confounders. The observational nature of our study means we cannot deduce the exact direction of effect of our findings. In addition, cognitive ability test scores in older people may reflect not only a possible decline, but also their peak prior cognitive ability; but we did not have any information regarding the trajectories of their cognitive function during the lifespan. Conclusion: Health practitioners should be encouraged to screen older men relating to their sexual activity to identify those who may be at risk of cognitive decline. Older men will be heartened to know that sexual activity may aid in the prevention of age-related decline in cognition.