Multidimensional Alternating Kernel Method for cortical layer segmentation in 3D reconstructed histology

The neocortex of the brain can be divided into six layers each with a distinct cell composition and connectivity pattern. Recently, sensory deprivation, including congenital deafness, has been shown to alter cortical structure (e.g. the cortical thickness) of the feline auditory cortex with variable and inconsistent results. Thus, understanding these complex changes will require further study of the constituent cortical layers in three-dimensional space. Further progress crucially depends on the use of objective computational techniques that can reliably characterize spatial properties of the complex cortical structure. Here a method for cortical laminar segmentation is derived and applied to the three-dimensional cortical areas reconstructed from a series of histological sections from four feline brains. In this approach, the Alternating Kernel Method was extended to fit a multi-variate Gaussian mixture model to a feature space consisting of both staining intensity and a biologically plausible equivolumetric depth map. This research method•Extends the Alternating Kernel Method to multi-dimensional feature spaces.•Uses it to segment the cortical layers in reconstructed histology volume. Segmentation features include staining intensity and a biologically plausible equivolumetric depth map.•Validates results in auditory cortical areas of feline brains, two with normal hearing and two with congenital deafness.