Considerable evidence from different methodologies has identified the anterior temporal lobes (ATLs) as key regions for the representation of semantic knowledge. Research interest is now shifting to investigate the roles of different ATL subregions in semantic representation, with particular emphasis on the functions of the left versus right ATLs. In this review, we provide evidence for graded specializations both between and within the ATLs. We argue (1) that multimodal, pan-category semantic representations are supported jointly by both left and right ATLs, yet (2) that the ATLs are not homogeneous in their function. Instead, subtle functional gradations both between and within the ATLs emerge as a consequence of differential connectivity with primary sensory/motor/limbic regions. This graded specialization account of semantic representation provides a compromise between theories that posit no differences between the functions of the left and right ATLs and those that posit that the left and right ATLs are entirely segregated in function. Evidence for this graded account comes from converging sources, and its benefits have been exemplified in formal computational models. We propose that this graded principle is not only a defining feature of the ATLs but is also a more general neurocomputational principle found throughout the temporal lobes.