Abstract:Repeated measures analyses of variance are the method of choice in many studies from experimental psychology and the neurosciences. Data from these fields are often characterized by small sample sizes, high numbers of factor levels of the within-subjects factor(s), and nonnormally distributed response variables such as response times. For a design with a single within-subjects factor, we investigated Type I error control in univariate tests with corrected degrees of freedom, the multivariate approach, and a mixed-model (multilevel) approach (SAS PROC MIXED) with Kenward–Roger’s adjusted degrees of freedom. We simulated multivariate normal and nonnormal distributions with varied population variance–covariance structures (spherical and nonspherical), sample sizes (N), and numbers of factor levels (K). For normally distributed data, as expected, the univariate approach with Huynh–Feldt correction controlled the Type I error rate with only very few exceptions, even if samples sizes as low as three were combined with high numbers of factor levels. The multivariate approach also controlled the Type I error rate, but it requires N ≥ K. PROC MIXED often showed acceptable control of the Type I error rate for normal data, but it also produced several liberal or conservative results. For nonnormal data, all of the procedures showed clear deviations from the nominal Type I error rate in many conditions, even for sample sizes greater than 50. Thus, none of these approaches can be considered robust if the response variable is nonnormally distributed. The results indicate that both the variance heterogeneity and covariance heterogeneity of the population covariance matrices affect the error rates.

Estimates of test size for several test procedures based on conventional variance ratios in the repeated measures design

Some approximate tests for repeated measurement designs

Evaluating the robustness of repeated measures analyses: The case of small sample sizes and nonnormal data

Tests for Covariance Structures with High-Dimensional Repeated Measurements

Impact of Sample Size and Variability on the Power and Type I Error Rates of Equivalence Tests: A Simulation Study.

Wald tests for variance-adjusted equivalence assessment with normal endpoints

Inference for Repeated Measures Models under Heteroscedasticity

Sample Size Determination for Testing the Variance Compound in a One-Way Random Effects Model

Power Analysis for Experiments with Clustered Data, Ratio Metrics, and Regression for Covariate Adjustment

Two tests of variance homogeneity for clustered data where group size is informative

Tests for equivalence based on odds ratio for matched-pair design

The analysis of repeated measurements: A quantitative research synthesis

A comparison of recent approaches to the analysis of repeated measurements

3-stage and 4-stage tests with deterministic stage sizes and non-iid data

RESEARCH ON ANALYSIS OF VARIANCE AND DATA INTERPRETATION

Testing two variances for superiority/non-inferiority and equivalence: Using the exhaustion algorithm for sample size allocation with cost

Testing the Equality of Proportions for Combined Unilateral and Bilateral Data

Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit?

An Adjustment to the Bartlett's Test for Small Sample Size.

Meta-analysis of ratios of sample variances

A Practitioner's Guide to Multiple Testing Error Rates