Abstract:Collaborative efforts to directly replicate empirical studies in the medical and social sciences have revealed alarmingly low rates of replicability, a phenomenon dubbed the 'replication crisis'. Poor replicability has spurred cultural changes targeted at improving reliability in these disciplines. Given the absence of equivalent replication projects in ecology and evolutionary biology, two inter-related indicators offer the opportunity to retrospectively assess replicability: publication bias and statistical power. This registered report assesses the prevalence and severity of small-study (i.e., smaller studies reporting larger effect sizes) and decline effects (i.e., effect sizes decreasing over time) across ecology and evolutionary biology using 87 meta-analyses comprising 4,250 primary studies and 17,638 effect sizes. Further, we estimate how publication bias might distort the estimation of effect sizes, statistical power, and errors in magnitude (Type M or exaggeration ratio) and sign (Type S). We show strong evidence for the pervasiveness of both small-study and decline effects in ecology and evolution. There was widespread prevalence of publication bias that resulted in meta-analytic means being over-estimated by (at least) 0.12 standard deviations. The prevalence of publication bias distorted confidence in meta-analytic results, with 66% of initially statistically significant meta-analytic means becoming non-significant after correcting for publication bias. Ecological and evolutionary studies consistently had low statistical power (15%) with a 4-fold exaggeration of effects on average (Type M error rates = 4.4). Notably, publication bias reduced power from 23% to 15% and increased type M error rates from 2.7 to 4.4 because it creates a non-random sample of effect size evidence. The sign errors of effect sizes (Type S error) increased from 5% to 8% because of publication bias. Our research provides clear evidence that many published ecological and evolutionary findings are inflated. Our results highlight the importance of designing high-power empirical studies (e.g., via collaborative team science), promoting and encouraging replication studies, testing and correcting for publication bias in meta-analyses, and adopting open and transparent research practices, such as (pre)registration, data- and code-sharing, and transparent reporting.

Publication bias in psychology: A closer look at the correlation between sample size and effect size

The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases

Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis

Estimating the change in meta-analytic effect size estimates after the application of publication bias adjustment methods.

Publication Bias in the Organizational Sciences

Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature

Footprint of publication selection bias on meta‐analyses in medicine, environmental sciences, psychology, and economics

Challenging the N-Heuristic: Effect size, not sample size, predicts the replicability of psychological science

An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports

Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology

Publication Bias in Special Education Meta-Analyses

Recent reproducibility estimates indicate that negative evidence is observed over 30 times before publication

Examining publication bias—a simulation-based evaluation of statistical tests on publication bias

Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods

Power to Detect What? Considerations for Planning and Evaluating Sample Size

Publication Bias and Dissemination of Clinical Research

Efficacy of cognitive–behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias

Correlational effect size benchmarks.

Sensitivity analysis for publication bias in meta‐analyses

Small Studies May Overestimate the Effect Sizes in Critical Care Meta-Analyses: A Meta-Epidemiological Study

Changes in methodological study characteristics in psychology between 2010-2021