Proton driven plasma wakefield acceleration features both ultra-high accelerating gradients thanks to plasmas and long acceleration distances due to high energy contents of protons. Hence it has great potential of powering leptons to TeV-scale energies in a single plasma stage. Nonetheless, the plasma response is different from that driven by the electron bunches or lasers as protons suck in the plasma electrons rather than repel them. It leads to radially nonlinear and time-varying focusing at the accelerating region, which is detrimental to the witness beam quality. In this thesis, by theory and particle-in-cell simulations, we demonstarte the viability of generation of high energy high quality electrons or positrons driven by a single short proton bunch or a proton bunch train, which paves the way for the development of future energy frontier lepton colliders. The hollow plasma channel has been involved into our studies, which plays a vital role in creating a beneficial accelerating structure. It eliminates the transverse plasma fields and forms radially uniform accelerating fields. The former benefits preservation of the beam emittance. The latter enables minimization of the energy spread by longitudinal control of the beam. The relating issues have been investigated, for example, the transverse effects induced by the beam-hollow misalignment, and the wakefield excitation driven by a practically long proton bunch in the uniform plasma, where the concept of seeded self-modulation has been employed into the AWAKE experiment.