Using cluster-based permutation tests to estimate MEG/EEG onsets: how bad is it?

Guillaume A. Rousselet

DOI: https://doi.org/10.1101/2023.11.13.566864

2024-09-12

Abstract:Localising effects in space, time and other dimensions is a fundamental goal of magneto- and electro-encephalography (EEG) research. A popular exploratory approach applies mass-univariate statistics followed by cluster-sum inferences, an effective way to correct for multiple comparisons while preserving high statistical power by pooling together neighbouring effects. However, these cluster-based methods have an important limitation: each cluster is associated with a unique value, such that there is no error control at individual time points, and one must be cautious about interpreting when and where effects start and end. provided an important reminder of this limitation. They also reported results from a simulation, suggesting that onsets estimated from EEG data are both positively biased and very variable. However, the simulation lacked comparisons to other methods. Here I report such comparisons in a new simulation, replicating the positive bias of the cluster-sum method, but also demonstrating that it performs relatively well, in terms of bias and variability, compared to other methods that provide point-wise values: two methods that control the false discovery rate, and two methods that control the family-wise error rate (cluster-depth and maximum statistic methods). I also present several strategies to reduce estimation bias, including group calibration, group comparison, and using binary segmentation, a simple change point detection algorithm that outperformed mass-univariate methods in simulations. Finally, I demonstrate how to generate onset confidence intervals that integrate variability over trials and participants, using a hierarchical bootstrap, a substantial improvement over standard group approaches that ignore measurement uncertainty.

Neuroscience