Identifying treatment effects in multi-channel measurements in electroencephalographic studies: Multivariate permutation tests and multiple comparisons

A versatile procedure is described comprising an application of statistical techniques to the analysis of the large, multi-dimensional data arrays produced by electroencephalographic (EEG) measurements of human brain function. Previous analytical methods have been unable to identify objectively the precise times at which statistically significant experimental effects occur, owing to the large number of variables (electrodes) and small number of subjects, or have been restricted to two-treatment experimental designs. Many time-points are sampled in each experimental trial, making adjustment for multiple comparisons mandatory. Given the typically large number of comparisons and the clear dependence structure among time-points, simple Bonferroni-type adjustments are far too conservative. A three-step approach is proposed: (i) summing univariate statistics across variables; (ii) using permutation tests for treatment effects at each time-point; and (iii) adjusting for multiple comparisons using permutation distributions to control family-wise error across the whole set of time-points. Our approach provides an exact test of the individual hypotheses while asymptotically controlling family-wise error in the strong sense, and can provide tests of interaction and main effects in factorial designs. An application to two experimental data sets from EEG studies is described, but the approach has application to the analysis of spatio-temporal multivariate data gathered in many other contexts.