Abstract:Meta‐analysis produces a quantitative synthesis of evidence‐based knowledge, shaping not only research trends but also policies and practices in biology. However, two statistical issues, selective reporting and statistical dependence, can severely distort meta‐analytic parameter estimation and inference. Here, we re‐analyse 448 meta‐analyses to demonstrate a new two‐step procedure to deal with two common challenges in biological meta‐analyses that often occur simultaneously: publication bias and non‐independence. First, we employ bias‐robust weighting schemes under the generalized least square estimator to obtain average effect sizes that are more robust to selective reporting. We then use cluster‐robust variance estimation to account for statistical dependence, reducing bias in estimating standard errors and ensuring valid statistical inference. The first step of our approach demonstrates comparable performance in estimating average effect sizes to the existing publication‐bias adjustment methods in the presence of selective reporting. This equivalence holds across two publication bias selection processes. The second step achieves estimates of standard errors consistent with the multilevel meta‐analytic model, a benchmark method with adequate control of Type I error rates for multiple, statistically dependent effect sizes. Re‐analyses of 448 meta‐analyses show that ignoring these two issues tends to overestimate effect sizes by an average of 110% and underestimate standard errors by 120%. To facilitate implementation, we have developed a website including a step‐by‐step tutorial. Complementing current meta‐analytic workflows with the proposed method as a sensitivity analysis can facilitate a transition to a more robust approach in quantitative evidence synthesis.

Robust inference methods for meta‐analysis involving influential outlying studies

Outlier detection and influence diagnostics in network meta-analysis

A Unified Method for Improved Inference in Random-effects Meta-analysis

Robust inference for the unification of confidence intervals in meta-analysis

Random-Effect Meta-Analysis with Robust Between-Study Variance

Robust Point and Variance Estimation for Meta‐analyses with Selective Reporting and Dependent Effect Sizes

Bayesian influence diagnostics and outlier detection for meta-analysis of diagnostic test accuracy

Robust Estimation and Confidence Interval in Meta-Regression Models

A confidence interval robust to publication bias for random-effects meta-analysis of few studies

Sensitivity analysis with iterative outlier detection for systematic reviews and meta‐analyses

A novel robust meta-analysis model using the t distribution for outlier accommodation and detection

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

Bayesian model-based outlier detection in network meta-analysis

Robust Confidence Intervals for Meta-Regression with Interaction Effects

Robust Bayesian meta-analysis: Addressing publication bias with model-averaging.

Small sample adjustments for robust variance estimation with meta-regression.

Robust Inference under the Beta Regression Model with Application to Health Care Studies

Meta-analysis with Robust Variance Estimation: Expanding the Range of Working Models

Meta-Analysis with Untrusted Data

Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study

A modified regression method to test publication bias in meta-analyses with binary outcomes