Nearly 50% of cancer patients undergo radiotherapy. Late radiotherapy toxicity affects quality-of-life in long-term cancer survivors and risk of side-effects in a minority limits doses prescribed to the majority of patients. Development of a test predicting risk of toxicity could benefit many cancer patients. We aimed to meta-analyze individual level data from four genome-wide association studies from prostate cancer radiotherapy cohorts including 1564 men to identify genetic markers of toxicity. Prospectively assessed two-year toxicity endpoints (urinary frequency, decreased urine stream, rectal bleeding, overall toxicity) and single nucleotide polymorphism (SNP) associations were tested using multivariable regression, adjusting for clinical and patient-related risk factors. A fixed-effects meta-analysis identified two SNPs: rs17599026 on 5q31.2 with urinary frequency (odds ratio [OR] 3.12, 95% confidence interval [CI] 2.08–4.69, p-value 4.16 × 10− 8) and rs7720298 on 5p15.2 with decreased urine stream (OR 2.71, 95% CI 1.90–3.86, p-value = 3.21 × 10− 8). These SNPs lie within genes that are expressed in tissues adversely affected by pelvic radiotherapy including bladder, kidney, rectum and small intestine. The results show that heterogeneous radiotherapy cohorts can be combined to identify new moderate-penetrance genetic variants associated with radiotherapy toxicity. The work provides a basis for larger collaborative efforts to identify enough variants for a future test involving polygenic risk profiling.