Cerebral small vessel disease (SVD) and inflammation are increasingly recognized as key contributors to Alzheimer's disease (AD), although the timing, trajectory, and relation between them early in the disease process is unclear. Therefore, to investigate very early-stage changes, we compared 158 healthy midlife adults with and without inherited AD predisposition (APOE4 carriership (38% positive), parental family history (FH) of dementia (54% positive)) on markers of SVD (white matter hyperintensities (WMH), cerebral microbleeds), and inflammation (C-reactive protein (CRP), fibrinogen), cross-sectionally and longitudinally over two years. While WMH severity was comparable between groups at baseline, longitudinal progression of WMH was greater in at-risk groups (APOE4+ and FH+). Topographically, APOE4 was associated exclusively with deep, but not periventricular, WMH progression after adjusting for FH. Conversely, APOE4 carriers displayed lower CRP levels than noncarriers, but not fibrinogen. Furthermore, interaction analysis showed that FH moderated the effect of SVD and inflammation on reaction time, an early feature of SVD, but not episodic memory or executive function. Findings suggest that vascular and inflammatory changes could occur decades before dementia onset, and may be of relevance in predicting incipient clinical progression.