This paper assesses the ability of smoothed particle hydrodynamics (SPH) to simulate mixing of two-phase flows and their transition to instabilities under different flow regimes. A new measure for quantification of the degree of mixing between phases in a Lagrangian framework is also developed. The method is validated using the lid-driven cavity and two-phase Poiseuille flow cases. The velocity along the centre of the cavity is compared with results from the literature, whilst commercial volume-of-fluid code STAR-CCM+ provides a benchmark for the mixing and different mixing measures are considered. The velocity of two-phase Poiseuille flow along the channel is compared to the analytical solution, and the appearance of interfacial instabilities with perturbation theory. This is the first time SPH has been used to investigate the onset and development of these instabilities. In particular, it is able to model the deforming shape of the interface, which is not given by analytical studies, while also offering improved predictions over conventional mesh-based computational fluid dynamics simulations.