An as yet outstanding goal in quantum optics is the realization of fast optical nonlinearities at the single-photon level. This would allow for the implementation of optical devices with new functionalities such as single-photon switches/transistors or controlled-phase gates. Although nonlinear optics effects at the single-emitter level have been demonstrated in a number of systems, none of these experiments showed single-photon switching on ultrafast timescales. Here, we perform pulsed two-colour spectroscopy and demonstrate that, in a strongly coupled quantum dot-cavity system, the presence of a single photon on one of the fundamental polariton transitions can turn on light scattering on a transition from the first to the second Jaynes-Cummings manifold. The overall switching time of this single-photon all-optical switch is ∼50 ps. In addition, we use the single-photon nonlinearity to implement a pulse correlator. Our quantum dot-cavity system could form the building block of future high-bandwidth photonic networks operating in the quantum regime.