The term “Primary age-related tauopathy” (PART) was coined in 2014 to describe the common neuropathological observation of neurofibrillary tangles without associated beta-amyloid pathology. It is possible for PART pathology to be present in both cognitively normal and cognitively impaired individuals. Genetically, Apolipoprotein E (APOE) ε4 has been shown to occur less commonly in PART than in Alzheimer’s disease (AD). Here, we investigate the relationships between PART, AD and those pathologically normal for age, with an emphasis on APOE and cognition, using 152 selected participants from The University of Manchester Longitudinal Study of Cognition in Normal Health Old Age and the Manchester arm of the Brains for Dementia Research cohort. APOE genotype differed between PART and AD with APOE ε2 more common in the former and APOE ε4 more common in the latter. Individuals with definite PART were less likely to be cognitively impaired than those with AD and those with pathology considered pathologically normal for age. We postulate that the lack of beta-amyloid (Aβ) in definite PART cases may be due either to an increased frequency of APOE ε2 or decreased frequency of APOE ε4 as their resulting protein isoforms have differing binding properties in relation to Aβ. Similarly, an increased frequency of APOE ε2 or decreased frequency of APOE ε4 may lead to decreased levels of cognitive impairment, which raises questions regarding the impact of Aβ pathology on overall cognition in elderly subjects. We suggest that it may be possible to use the increased frequency of APOE ε2 in definite PART to assist neuropathological diagnosis.