We show that the primary cause of errors in a broad class of optical quantum-logic gates are due to the higher-order photon terms in parametric downconversion sources. A model describing real-life imperfections in these entangling gates is presented and tested in an experiment where we entangle dependent photons from the same downconversion source using a controlled-z gate, and measure the state tomographically. We find good agreement between the modelled and measured results. Our investigations demonstrate that, although small, these noise terms are amplified by the intrinsic non-determinism of the gates. It is worth considering alternative schemes based on weak nonlinearities to see if they are more resilient to this degradation.