Abstract:Meta‐analysis produces a quantitative synthesis of evidence‐based knowledge, shaping not only research trends but also policies and practices in biology. However, two statistical issues, selective reporting and statistical dependence, can severely distort meta‐analytic parameter estimation and inference. Here, we re‐analyse 448 meta‐analyses to demonstrate a new two‐step procedure to deal with two common challenges in biological meta‐analyses that often occur simultaneously: publication bias and non‐independence. First, we employ bias‐robust weighting schemes under the generalized least square estimator to obtain average effect sizes that are more robust to selective reporting. We then use cluster‐robust variance estimation to account for statistical dependence, reducing bias in estimating standard errors and ensuring valid statistical inference. The first step of our approach demonstrates comparable performance in estimating average effect sizes to the existing publication‐bias adjustment methods in the presence of selective reporting. This equivalence holds across two publication bias selection processes. The second step achieves estimates of standard errors consistent with the multilevel meta‐analytic model, a benchmark method with adequate control of Type I error rates for multiple, statistically dependent effect sizes. Re‐analyses of 448 meta‐analyses show that ignoring these two issues tends to overestimate effect sizes by an average of 110% and underestimate standard errors by 120%. To facilitate implementation, we have developed a website including a step‐by‐step tutorial. Complementing current meta‐analytic workflows with the proposed method as a sensitivity analysis can facilitate a transition to a more robust approach in quantitative evidence synthesis.

P‐hacking in meta‐analyses: A formalization and new meta‐analytic methods

Sensitivity analysis for publication bias in meta‐analyses

The Extent and Consequences of P-Hacking in Science

Adjusting for Publication Bias in JASP and R: Selection Models, PET-PEESE, and Robust Bayesian Meta-Analysis

Robust Point and Variance Estimation for Meta‐analyses with Selective Reporting and Dependent Effect Sizes

Detecting p-hacking

Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use

Investigating and dealing with publication bias and other reporting biases in meta‐analyses of health research: A review

A modified regression method to test publication bias in meta-analyses with binary outcomes

Using Monte Carlo experiments to select meta‐analytic estimators

Methods to Address Confounding and Other Biases in Meta-Analyses: Review and Recommendations

Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods

Hybrid test for publication bias in meta-analysis

Estimating the change in meta-analytic effect size estimates after the application of publication bias adjustment methods.

Robust Bayesian meta-analysis: Addressing publication bias with model-averaging.

Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment

Estimating the extent of selective reporting: An application to economics

P Value–driven Methods Were Underpowered to Detect Publication Bias: Analysis of Cochrane Review Meta-Analyses

Meta-Analysis with Untrusted Data

Addressing researcher degrees of freedom through minP adjustment

A re-analysis of about 60,000 sparse data meta-analyses suggests that using an adequate method for pooling matters