Polymer nanocomposites reinforced with carbon-based nanofillers are gaining increasing interest for a number of applications due to their excellent properties. The understanding of the reinforcing mechanisms is, therefore, very important for the maximization of performance. This present review summarizes the current literature status on the mechanical properties of composites reinforced with graphene-related materials (GRMs) and carbon nanotubes (CNTs) and identifies the parameters that clearly affect the mechanical properties of the final materials. It is also shown how Raman spectroscopy can be utilized for the understanding of the stress transfer efficiency from the matrix to the reinforcement and it can even be used to map stress and strain in graphene. Importantly, it is demonstrated clearly that continuum micromechanics that was initially developed for fibre-reinforced composites is still applicable at the nanoscale for both GRMs and CNTs. Finally, current problems and future perspectives are discussed.