Abstract:Regressions and meta-regressions are widely used to estimate patterns and effect sizes in various disciplines. However, many biological and medical analyses use relatively low sample size (N), contributing to concerns on reproducibility. What is the minimum N to identify the most plausible data pattern using regressions? Statistical power analysis is often used to answer that question, but it has its own problems and logically should follow model selection to first identify the most plausible model. Here we make null, simple linear and quadratic data with different variances and effect sizes. We then sample and use information theoretic model selection to evaluate minimum N for regression models. We also evaluate the use of coefficient of determination (R2) for this purpose; it is widely used but not recommended. With very low variance, both false positives and false negatives occurred at N < 8, but data shape was always clearly identified at N ≥ 8. With high variance, accurate inference was stable at N ≥ 25. Those outcomes were consistent at different effect sizes. Akaike Information Criterion weights (AICc wi) were essential to clearly identify patterns (e.g., simple linear vs. null); R2 or adjusted R2 values were not useful. We conclude that a minimum N = 8 is informative given very little variance, but minimum N ≥ 25 is required for more variance. Alternative models are better compared using information theory indices such as AIC but not R2 or adjusted R2. Insufficient N and R2-based model selection apparently contribute to confusion and low reproducibility in various disciplines. To avoid those problems, we recommend that research based on regressions or meta-regressions use N ≥ 25.

Minimal sample size in balanced ANOVA models of crossed, nested, and mixed classifications

Tests for a Multiple-Sample Problem in High Dimensions

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Power Analysis and Sample Size Planning in ANCOVA Designs

Tests under simple order in one‐way ANCOVA

Interaction Balance for Symmetrical Factorial Designs with Generalized Minimum Aberration

Alternate Forms of the One-Way ANOVA F and Kruskal–Wallis Test Statistics

Optimal sample size allocation for Welch’s test in one-way heteroscedastic ANOVA

Nearly Resolution V Plans on Blocks of Small Size

Efficient Balanced Treatment Assignments for Experimentation

Approximating Bayes factors from minimal ANOVA summaries: An extension of the BIC method

A solution to minimum sample size for regressions

Multivariate nonparametric methods in two-way balanced designs: performances and limitations in small samples

Permutation Strategies for Inference in ANOVA‐Based Models for Nonorthogonal Designs Including Continuous Covariates

Minimizing Sample Size When Using Exploratory Factor Analysis for Measurement

A simple sample size formula for analysis of covariance in cluster randomized trials

Optimal allocation of sample size for randomization-based inference from 2 K factorial designs

Type I Error Robustness of ANOVA and ANOVA on Ranks When the Number of Treatments is Large

Sample Size Determination for Testing the Variance Compound in a One-Way Random Effects Model

Rank-Based Procedures in Factorial Designs: Hypotheses about Nonparametric Treatment Effects

Simulation-Based Power-Analysis for Factorial ANOVA Designs