The aim of this paper is to determine what non-exposure factors influence the relationship between vibration and noise exposure from the construction of a Light Rapid Transit (LRT) system and the annoyance of nearby residents. Noise and vibration from construction sites are known to annoy residents, with annoyance increasing as a function of the magnitude of the vibration and noise. There is not a strong correlation between exposure and levels of annoyance suggesting that factors not directly related to the exposure may have an influence. A range of attitudinal, situational and demographic factors are investigated with the aim of understanding the wide variation in annoyance for a given vibration exposure. A face-to-face survey of residents (n = 350) near three sites of LRT construction was conducted, and responses were compared to semi-empirical estimates of the internal vibration within the buildings. It was found that annoyance responses due to vibration were strongly influenced by two attitudinal variables, concern about property damage and sensitivity to vibration. Age, ownership of the property and the visibility of the construction site were also important factors. Gender, time at home and expectation of future levels of vibration had much less influence. Due to the measurement methods used, it was not possible to separate out the effects of noise and vibration on annoyance; as such, this paper focusses on annoyance due to vibration exposure. This work concludes that for the most cost-effective reduction of the impact of construction vibration and noise on the annoyance felt by a community, policies should consider attitudinal factors.