Abstract:Researchers and decision‐makers often use evidence from randomised controlled trials (RCTs) to determine the efficacy or effectiveness of a treatment or intervention. Studies with observational designs are often used to measure the effectiveness of an intervention in 'real world' scenarios. Numerous study designs and their modifications (including both randomised and observational designs) are used for comparative effectiveness research in an attempt to give an unbiased estimate of whether one treatment is more effective or safer than another for a particular population. An up‐to‐date systematic analysis is needed to identify differences in effect estimates from RCTs and observational studies. This updated review summarises the results of methodological reviews that compared the effect estimates of observational studies with RCTs from evidence syntheses that addressed the same health research question. To assess and compare synthesised effect estimates by study type, contrasting RCTs with observational studies. To explore factors that might explain differences in synthesised effect estimates from RCTs versus observational studies (e.g. heterogeneity, type of observational study design, type of intervention, and use of propensity score adjustment). To identify gaps in the existing research comparing effect estimates across different study types. We searched MEDLINE, the Cochrane Database of Systematic Reviews, Web of Science databases, and Epistemonikos to May 2022. We checked references, conducted citation searches, and contacted review authors to identify additional reviews. We included systematic methodological reviews that compared quantitative effect estimates measuring the efficacy or effectiveness of interventions tested in RCTs versus in observational studies. The included reviews compared RCTs to observational studies (including retrospective and prospective cohort, case‐control and cross‐sectional designs). Reviews were not eligible if they compared RCTs with studies that had used some form of concurrent allocation. Using results from observational studies as the reference group, we examined the relative summary effect estimates (risk ratios (RRs), odds ratios (ORs), hazard ratios (HRs), mean differences (MDs), and standardised mean differences (SMDs)) to evaluate whether there was a relatively larger or smaller effect in the ratio of odds ratios (ROR) or ratio of risk ratios (RRR), ratio of hazard ratios (RHR), and difference in (standardised) mean differences (D(S)MD). If an included review did not provide an estimate comparing results from RCTs with observational studies, we generated one by pooling the estimates for observational studies and RCTs, respectively. Across all reviews, we synthesised these ratios to produce a pooled ratio of ratios comparing effect estimates from RCTs with those from observational studies. In overviews of reviews, we estimated the ROR or RRR for each overview using observational studies as the reference category. We appraised the risk of bias in the included reviews (using nine criteria in total). To receive an overall low risk of bias rating, an included review needed: explicit criteria for study selection, a complete sample of studies, and to have controlled for study methodological differences and study heterogeneity. We assessed reviews/overviews not meeting these four criteria as having an overall high risk of bias. We assessed the certainty of the evidence, consisting of multiple evidence syntheses, with the GRADE approach. We included 39 systematic reviews and eight overviews of reviews, for a total of 47. Thirty‐four of these contributed data to our primary analysis. Based on the available data, we found that the reviews/overviews included 2869 RCTs involving 3,882,115 participants, and 3924 observational studies with 19,499,970 participants. We rated 11 reviews/overviews as having an overall low risk of bias, and 36 as having an unclear or high risk of bias. Our main concerns with the included reviews/overviews were that some did not assess the quality of their included studies, and some failed to account appropriately for differences between study designs – for example, they conducted aggregate analyses of all observational studies rather than separate analyses of cohort and case‐control studies. When pooling RORs and RRRs, the ratio of ratios indicated no difference or a very small difference between the effect estimates from RCTs versus from observational studies (ratio of ratios 1.08, 95% confidence interval (CI) 1.01 to 1.15). We rated the certainty of the evidence as low. Twenty‐three of 34 reviews reported effect estimates of RCTs and observational studies that were on average in agreement. In a number of subgroup analyses, small differences in the effect estimates were detected: ‐ pharmaceutical interventions only (ratio of ratios 1.12, 95% CI 1.04 to 1.21); ‐ RCTs and observational -Abstract Truncated-

Observational studies using propensity score analysis underestimated the effect sizes in critical care medicine.

Do the observational studies using propensity score analysis agree with randomized controlled trials in the area of sepsis?

Small Studies May Overestimate the Effect Sizes in Critical Care Meta-Analyses: A Meta-Epidemiological Study

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study

The efficacy of activated protein C for the treatment of sepsis: incorporating observational evidence with a Bayesian approach.

Recombinant Human Activated Protein C for the Treatment of Severe Sepsis and Septic Shock: a Study Protocol for Incorporating Observational Evidence Using a Bayesian Approach.

Propensity Score Methods for Creating Covariate Balance in Observational Studies

Associations Between Elevated Systolic Blood Pressure and Outcomes in Critically Ill Patients: A Retrospective Cohort Study and Propensity Analysis

Comparing Propensity Score-Based Methods in Estimating the Treatment Effects: A Simulation Study

Evaluation of the proportional hazards assumption and covariate adjustment methods in comparative surgical observational studies with time-to-event endpoints

Use and Interpretation of Propensity Scores in Aging Research: A Guide for Clinical Researchers

Comparison of statistical methods for the analysis of patient-reported outcomes in randomised controlled trials: A simulation study

Five Steps to Successfully Implement and Evaluate Propensity Score Matching in Clinical Research Studies

An empirical comparison of the harmful effects for randomized controlled trials and non-randomized studies of interventions

[Application of Propensity Score Matching in the Design of an Epidemiological Study].

Compelling evidence from meta-epidemiological studies demonstrates overestimation of effects in randomized trials that fail to optimize randomization and blind patients and outcome assessors

Comparison of treatment effect estimates of non-vitamin K antagonist oral anticoagulants versus warfarin between observational studies using propensity score methods and randomized controlled trials

Use of Propensity Scoring and Its Application to Real‐World Data: Advantages, Disadvantages, and Methodological Objectives Explained to Researchers Without Using Mathematical Equations

Effectiveness of treatment based on PiCCO parameters in critically ill patients with septic shock and/or acute respiratory distress syndrome: a randomized controlled trial

Evaluation of the Propensity score methods for estimating marginal odds ratios in case of small sample size

Journal impact factor, trial effect size, and methodological quality appear scantly related: a systematic review and meta-analysis