We make use of interferometric CO and Hi observations, and optical integral-field spectroscopy from the ATLAS3D survey, to probe the origin of the molecular and ionized interstellar medium (ISM) in local early-type galaxies. We find that 36 ± 5 per cent of our sample of fast-rotating early-type galaxies have their ionized gas kinematically misaligned with respect to the stars, setting a strong lower limit on the importance of externally acquired gas (e.g. from mergers and cold accretion). Slow rotators have a flat distribution of misalignments, indicating that the dominant source of gas is external. The molecular, ionized and atomic gas in all the detected galaxies are always kinematically aligned, even when they are misaligned from the stars, suggesting that all these three phases of the ISM share a common origin. In addition, we find that the origin of the cold and warm gas in fast-rotating early-type galaxies is strongly affected by environment, despite the molecular gas detection rate and mass fractions being fairly independent of group/cluster membership. Galaxies in dense groups and the Virgo cluster nearly always have their molecular gas kinematically aligned with the stellar kinematics, consistent with a purely internal origin (presumably stellar mass loss). In the field, however, kinematic misalignments between the stellar and gaseous components indicate that at least 42 ± 5 per cent of local fast-rotating early-type galaxies have their gas supplied from external sources. When one also considers evidence of accretion present in the galaxies' atomic gas distributions, ≳46 per cent of fast-rotating field ETGs are likely to have acquired a detectable amount of ISM from accretion and mergers. We discuss several scenarios which could explain the environmental dichotomy, including preprocessing in galaxy groups/cluster outskirts and the morphological transformation of spiral galaxies, but we find it difficult to simultaneously explain the kinematic misalignment difference and the constant detection rate. Furthermore, our results suggest that galaxy mass may be an important independent factor associated with the origin of the gas, with the most massive fast-rotating galaxies in our sample (MK≲-24mag; stellar mass of ≈8 × 1010 M⊙) always having kinematically aligned gas. This mass dependence appears to be independent of environment, suggesting it is caused by a separate physical mechanism.