Abstract:Meta‐analysis produces a quantitative synthesis of evidence‐based knowledge, shaping not only research trends but also policies and practices in biology. However, two statistical issues, selective reporting and statistical dependence, can severely distort meta‐analytic parameter estimation and inference. Here, we re‐analyse 448 meta‐analyses to demonstrate a new two‐step procedure to deal with two common challenges in biological meta‐analyses that often occur simultaneously: publication bias and non‐independence. First, we employ bias‐robust weighting schemes under the generalized least square estimator to obtain average effect sizes that are more robust to selective reporting. We then use cluster‐robust variance estimation to account for statistical dependence, reducing bias in estimating standard errors and ensuring valid statistical inference. The first step of our approach demonstrates comparable performance in estimating average effect sizes to the existing publication‐bias adjustment methods in the presence of selective reporting. This equivalence holds across two publication bias selection processes. The second step achieves estimates of standard errors consistent with the multilevel meta‐analytic model, a benchmark method with adequate control of Type I error rates for multiple, statistically dependent effect sizes. Re‐analyses of 448 meta‐analyses show that ignoring these two issues tends to overestimate effect sizes by an average of 110% and underestimate standard errors by 120%. To facilitate implementation, we have developed a website including a step‐by‐step tutorial. Complementing current meta‐analytic workflows with the proposed method as a sensitivity analysis can facilitate a transition to a more robust approach in quantitative evidence synthesis.

Using Monte Carlo experiments to select meta‐analytic estimators

Mostly Harmless Simulations? Using Monte Carlo Studies for Estimator Selection

Monte Carlo algorithms for model assessment via conflicting summaries

A comparison of methods for model selection when estimating individual treatment effects

Meta‐epidemiological Study: a Step‐by‐step Approach by Using R

Empirical Comparison of Publication Bias Tests in Meta-Analysis

Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods

A Comparison of Metaheuristic Optimization Algorithms for Scale Short-Form Development

Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use

[Selection Models and the File Drawer Problem]: Comment

Footprint of publication selection bias on meta‐analyses in medicine, environmental sciences, psychology, and economics

Uncertain about uncertainty in matching‐adjusted indirect comparisons? A simulation study to compare methods for variance estimation

Optimized Model Selection for Estimating Treatment Effects from Costly Simulations of the US Opioid Epidemic

Mathematical programming tools for randomization purposes in small two‐arm clinical trials: A case study with real data

Mathematical programming tools for randomization purposes in small two-arm clinical trials: A case study with real data

A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study

Robust Point and Variance Estimation for Meta‐analyses with Selective Reporting and Dependent Effect Sizes

Systematically Missing Data in Causally Interpretable Meta-Analysis

Evaluating Small Sample Approaches for Model Test Statistics in Structural Equation Modeling

A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis