Repeated overstimulation of the stress response system, caused by exposure to prolonged highly stressful experiences, is thought to affect brain structure, cognitive ability, and mental health. We tested the effects of highly stressful experiences during childhood and adulthood using data from the UK Biobank, a large-scale national health and biomedical study with over 500,000 participants. To do this, we defined four groups with high or low levels of childhood and/or adulthood stress. We then used T1-and diffusion-weighted MRI data to assess the macrostructure of grey matter and microstructure of white matter within limbic brain regions, commonly associated with the stress response. We also compared executive function and working memory between these groups. Our findings suggest that in females, higher levels of Childhood stress were associated with reduced connectivity within the posterior thalamic radiation and cingulum of the hippocampus. In males however, higher levels of Adulthood stress is associated with similar changes in brain microstructure in the posterior thalamic radiation and cingulum of the hippocampus. High stress in Childhood and Adulthood was associated with decreases in executive function and working memory in both males and females. Stress across the lifespan was also positively associated with the number of diagnosed mental health problems, with a stronger effect in females than in males. Finally, our findings also suggest that cognitive and mental health outcomes due to stress may be mediated by the sex specific stress related changes in brain microstructure. Together our findings demonstrate clear links between stress at distinct phases of the lifespan, changes in measures of brain microstructure, impairments in cognitive abilities and negative mental health outcomes.