In this paper a pseudo-spectral incompressible smoothed particle hydrodynamics (FFT-ISPH) solver is presented. While the solution of the linear system arising from the pressure Poisson equation in physical space using iterative solvers in incompressible SPH is a viable solution, it is widely accepted that the computational cost of solving the pressure Poisson equation by iterative solvers is excessive and affects negatively on the efficiency of the solver. The proposed scheme is an intermediate between a fully spectral and a standard incompressible SPH solver. Herein, the solution of the pressure Poisson equation is performed in spectral space whereas the discretisation and time integration are performed in physical space. This results in a second and higher order scheme with the classical first order projection and a third order Runge-Kutta time integration scheme, respectively. A detailed performance analysis shows gains in computational cost of two orders of magnitude. Further, it is demonstrated that the solution of the pressure Poisson equation in spectral space is independent of the number of neighbouring nodes, in contrast to the iterative solver whose cost increases by a factor of three with smoothing length to particle size ratio. Periodic, bounded, and mixed boundary conditions test cases have been used to demonstrate the applicability, accuracy and robustness of the scheme with second and fourth order convergence rates. The scheme opens an avenue for simulations of high order incompressible flows in SPH where filtering operations in the frequency domain can be performed straightforwardly.