Linear optics (LO) is a promising candidate for the implementation of quantum information processing protocols. In such systems, single photons are used to represent qubits. In practice, single photons from different sources will not be perfectly temporally and frequency matched. Therefore, understanding the effects of temporal and frequency mismatch is important in characterizing the dynamics of the system. In this paper, we discuss the impact of temporal and frequency mismatch, how they differ from each other and what their effect is on a simple LO quantum gate. We show that temporal and frequency mismatch have inherently different effects on the operation of the gate. We also consider the spectral effects of the photodetectors, focusing on time-resolved detection, which we show has a strong impact on the operation of such protocols.