Bispectrum phase, closure phase and their generalization to kernel phase are all independent of pupil-plane phase errors to first order. This property, when used with sparse aperture masking behind adaptive optics, has been used recently in high-contrast observations at or inside the formal diffraction limit of large telescopes. Finding the limitations to these techniques requires an understanding of spatial and temporal third-order phase effects, as well as effects such as time-variable dispersion when coupled with the non-zero bandwidths in real observations. In this paper, formulae describing many of these errors are developed, so that a comparison can be made to fundamental noise processes of photon noise and background noise. I show that the current generation of aperture-masking observations of young solar-type stars, taken carefully in excellent observing conditions, are consistent with being limited by temporal phase noise and photon noise. This has relevance for plans to combine pupil remapping with spatial filtering. Finally, I describe calibration strategies for kernel phase, including the optimized calibrator weighting as used for LkCa15, and the restricted kernel phase POISE (phase observationally independent of systematic errors) technique that avoids explicit dependence on calibrators.