Abstract:Objectives: To assess to what extent the overall quality of evidence indicates changes to observe intervention effect estimates when new data become available. Methods: We conducted a meta-epidemiological study. We obtained evidence from meta-analyses of randomized trials of Cochrane reviews addressing the same health-care question that was updated with inclusion of additional data between January 2016 and May 2021. We extracted the reported effect estimates with 95% confidence intervals (CIs) from meta-analyses and corresponding GRADE (Grading of Recommendations Assessment, Development, and Evaluation) assessments of any intervention comparison for the primary outcome in the first and the last updated review version. We considered the reported overall quality (certainty) of evidence (CoE) and specific evidence limitations (no, serious or very serious for risk of bias, imprecision, inconsistency, and/or indirectness). We assessed the change in pooled effect estimates between the original and updated evidence using the ratio of odds ratio (ROR), absolute ratio of odds ratio (aROR), ratio of standard errors (RoSE), direction of effects, and level of statistical significance. Results: High CoE without limitations characterized 19.3% (n = 29) out of 150 included original Cochrane reviews. The update with additional data did not systematically change the effect estimates (mean ROR 1.00; 95% CI 0.99-1.02), which deviated 1.06-fold from the older estimates (median aROR; interquartile range [IQR]: 1.01-1.15), gained precision (median RoSE 0.87; IQR 0.76-1.00), and maintained the same direction with the same level of statistical significance in 93% (27 of 29) of cases. Lower CoE with limitations characterized 121 original reviews and graded as moderate CoE in 30.0% (45 of 150), low CoE in 32.0% (48 of 150), and very low CoE in 18.7% (28 of 150) reviews. Their update had larger absolute deviations (median aROR 1.12 to 1.33) and larger gains in precision (median RoSE 0.78-0.86) without clear and consistent differences between these categories of CoE. Changes in effect direction or statistical significance were also more common in the lower quality evidence, again with a similar extent across categories (without change in 75.6%, 64.6%, and 75.0% for moderate, low, very low CoE). As limitations increased, effect estimates deviated more (aROR 1.05 with zero, 1.11 with one, 1.25 with two, 1.24 with three limitations) and changes in direction or significance became more frequent (93.2% stable with no limitations, 74.5% with one, 68.2% with two, and 61.5% with three limitations). Conclusion: High-quality evidence without methodological deficiencies is trustworthy and stable, providing reliable intervention effect estimates when updated with new data. Evidence of moderate and lower quality may be equally prone to being unstable and cannot indicate if available effect estimates are true, exaggerated, or underestimated.

The pertussis epidemic: informing strategies for prevention of severe disease

GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence—An overview in the context of health decision-making

How do network meta-analyses address intransitivity when assessing certainty of evidence: a systematic survey

GRADE pearls and pitfalls—Part 1: Systematic reviews and meta‐analyses

Network meta-analysis: a powerful tool for clinicians, decision-makers, and methodologists

Contribution of magnetostriction to transformer noise

Visualizing the assumptions of network meta‐analysis

A review and comparison of arm-based versus contrast-based network meta-analysis for binary outcomes-Understanding their differences and limitations

GRADE Notes 2: Criteria for searching non-randomized or indirect evidence should be defined early in the guideline production process

High certainty evidence is stable and trustworthy, whereas evidence of moderate or lower certainty may be equally prone to being unstable

What is the probability that higher versus lower quality of evidence represents true effects estimates?

A dialogue-based (natural language) platform for human interaction with an intelligent mining machine

GRADE guidance 23: Considering cost-effectiveness evidence in moving from evidence to health-related recommendations

GRADE pearls and pitfalls—Part 2: Clinical practice guidelines

Considerations Regarding a Network Meta-analysis of Systemic Treatments for Metastatic Castration-Sensitive Prostate Cancer—Reply

Interpretation of meta-analyses

A comment-driven evidence appraisal approach for decision-making when only uncertain evidence available

A modeling approach to derive baseline risk estimates for GRADE recommendations: Concepts, development, and results of its application to the American Society of Hematology 2019 guidelines on prevention of venous thromboembolism in surgical hospitalized patients

Optimized NH/sub 3/ annealing Process for high-quality HfSiON gate oxide

Twenty years of network meta‐analysis: Continuing controversies and recent developments

Characteristics of the sources, evaluation, and grading of the certainty of evidence in systematic reviews in public health: A methodological study