The key question of how the brain codes the meaning of words and pictures is the focus of vigorous debate. Is there a "semantic hub" in the temporal poles where these different inputs converge to form amodal conceptual representations? Alternatively, are there distinct neural circuits that underpin our comprehension of pictures and words? Understanding words might be primarily left-lateralised, linked to other language areas, while semantic representation of pictures may be more bilateral. To elucidate this debate, we used offline, low-frequency, repetitive transcranial magnetic stimulation (rTMS) to disrupt neural processing temporarily in the left or right temporal poles. During the induced refractory period, participants made judgements of semantic association for verbal and pictorial stimuli. The efficiency of semantic processing was reduced by rTMS, yet a perceptual task of comparable difficulty was unaffected. rTMS applied to the left or right temporal poles disrupted semantic processing for words and pictures to the same degree, while rTMS delivered at a control site had no impact. The results confirm that both temporal poles form a critical substrate within the neural network that supports conceptual knowledge, regardless of modality.