Ventilator-associated pneumonia is a common healthcare-associated infection with significant mortality, morbidity and healthcare cost, and rates have been proposed as a potential quality indicator. We examined ventilator-associated pneumonia rates as determined by different diagnostic scoring systems across four adult intensive care units in the North West of England. We also collected clinical opinions as to whether patients had ventilator-associated pneumonia, and whether patients were receiving antibiotics as treatment. Pooled ventilator-associated pneumonia rates were 36.3, 22.2, 15.2 and 1.1 per 1000 ventilator-bed days depending on the scoring system used. There was significant within-unit heterogeneity for ventilator-associated pneumonia rates calculated by the various scoring systems (all p < 0.001). Clinical opinion and antibiotic use did not correlate well with the scoring systems (k = 0.23 and k = 0.17, respectively). We therefore question whether the ventilator-associated pneumonia rate as measured by existing tools is either useful or desirable as a quality indicator.