Abstract:Repeated measures analyses of variance are the method of choice in many studies from experimental psychology and the neurosciences. Data from these fields are often characterized by small sample sizes, high numbers of factor levels of the within-subjects factor(s), and nonnormally distributed response variables such as response times. For a design with a single within-subjects factor, we investigated Type I error control in univariate tests with corrected degrees of freedom, the multivariate approach, and a mixed-model (multilevel) approach (SAS PROC MIXED) with Kenward–Roger’s adjusted degrees of freedom. We simulated multivariate normal and nonnormal distributions with varied population variance–covariance structures (spherical and nonspherical), sample sizes (N), and numbers of factor levels (K). For normally distributed data, as expected, the univariate approach with Huynh–Feldt correction controlled the Type I error rate with only very few exceptions, even if samples sizes as low as three were combined with high numbers of factor levels. The multivariate approach also controlled the Type I error rate, but it requires N ≥ K. PROC MIXED often showed acceptable control of the Type I error rate for normal data, but it also produced several liberal or conservative results. For nonnormal data, all of the procedures showed clear deviations from the nominal Type I error rate in many conditions, even for sample sizes greater than 50. Thus, none of these approaches can be considered robust if the response variable is nonnormally distributed. The results indicate that both the variance heterogeneity and covariance heterogeneity of the population covariance matrices affect the error rates.

Controlling the maximum familywise Type I error rate in analyses of multivariate experiments.

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Familywise Error Rate Controlling Procedures for Discrete Data

Power-enhanced multiple decision functions controlling family-wise error and false discovery rates

On limiting behaviors of stepwise multiple testing procedures

Familywise error control in multi-armed response-adaptive trials

On Asymptotic Behaviors of Stepwise Multiple Testing Procedures

Bayesian Multivariate Probability of Success Using Historical Data with Strict Control of Family-wise Error Rate

Evaluating the robustness of repeated measures analyses: The case of small sample sizes and nonnormal data

Addressing researcher degrees of freedom through minP adjustment

On stepdown control of the false discovery proportion

Combination rules for homoscedastic and heteroscedastic MANOVA models from multiply imputed datasets

Differences of Type I error rates for ANOVA and Multilevel-Linear-Models using SAS and SPSS for repeated measures designs

Analysis of error control in large scale two-stage multiple hypothesis testing

On Controlling the False Discovery Rate in Multiple Testing of the Means of Correlated Normals Against Two-Sided Alternatives

A Monte Carlo Study of the Effects of Common Method Variance on Significance Testing and Parameter Bias in Hierarchical Linear Modeling

Modifying the false discovery rate procedure based on the information theory under arbitrary correlation structure and its performance in high-dimensional genomic data

Optimal control of false discovery criteria in the two-group model

Covariate Adaptive Family-wise Error Rate Control for Genome-Wide Association Studies

MANOVA: A Procedure Whose Time Has Passed?

Making Meaning out of MANOVA: The Need for Multivariate Post Hoc Testing in Gifted Education Research