Abstract:Regressions and meta-regressions are widely used to estimate patterns and effect sizes in various disciplines. However, many biological and medical analyses use relatively low sample size (N), contributing to concerns on reproducibility. What is the minimum N to identify the most plausible data pattern using regressions? Statistical power analysis is often used to answer that question, but it has its own problems and logically should follow model selection to first identify the most plausible model. Here we make null, simple linear and quadratic data with different variances and effect sizes. We then sample and use information theoretic model selection to evaluate minimum N for regression models. We also evaluate the use of coefficient of determination (R2) for this purpose; it is widely used but not recommended. With very low variance, both false positives and false negatives occurred at N < 8, but data shape was always clearly identified at N ≥ 8. With high variance, accurate inference was stable at N ≥ 25. Those outcomes were consistent at different effect sizes. Akaike Information Criterion weights (AICc wi) were essential to clearly identify patterns (e.g., simple linear vs. null); R2 or adjusted R2 values were not useful. We conclude that a minimum N = 8 is informative given very little variance, but minimum N ≥ 25 is required for more variance. Alternative models are better compared using information theory indices such as AIC but not R2 or adjusted R2. Insufficient N and R2-based model selection apparently contribute to confusion and low reproducibility in various disciplines. To avoid those problems, we recommend that research based on regressions or meta-regressions use N ≥ 25.

Sample Size Determination for Testing the Variance Compound in a One-Way Random Effects Model

Review of sample size determination methods for the intraclass correlation coefficient in the one-way analysis of variance model

Determining Sample Size in Binary Measurement System

Sample Size Selection under an Infill Asymptotic Domain

Sample Size Calculation for Comparing Variabilities

Testing two variances for superiority/non-inferiority and equivalence: Using the exhaustion algorithm for sample size allocation with cost

Determining sample size adequacy for animal model studies in nutrition research: limits and ethical challenges of ordinary power calculation procedures

Estimates of test size for several test procedures based on conventional variance ratios in the repeated measures design

Optimal sample size allocation for Welch’s test in one-way heteroscedastic ANOVA

Impact of Sample Size and Variability on the Power and Type I Error Rates of Equivalence Tests: A Simulation Study.

Sample Size Estimation for Random-effects Models: Balancing Precision and Feasibility in Panel Studies

On Power and Sample Size Computation for Multiple Testing Procedures

A solution to minimum sample size for regressions

Small sample sizes: A big data problem in high-dimensional data analysis

Sample size determination for logistic regression revisited

Optimal allocation of sample size for randomization-based inference from 2 K factorial designs

A simple sample size formula for analysis of covariance in cluster randomized trials

Sample size determination for external pilot cluster randomised trials with binary feasibility outcomes: a tutorial

A new sample-size planning approach for person-specific VAR(1) studies: Predictive accuracy analysis

Assurance for sample size determination in reliability demonstration testing

Comparison of Software Algorithms for Calculating Reml Wald Type Confidence Limits for the Between-Group Variance Component in A Small Sample One-Way Random Effects Model Example