Abstract:Importance: Randomized clinical trials (RCTs) are widely regarded as the methodological benchmark for assessing clinical efficacy and safety of health interventions. There is growing interest in using nonrandomized studies to assess efficacy and safety of new drugs. Objective: To determine how treatment effects for the same drug compare when evaluated in nonrandomized vs randomized studies. Data sources: Meta-analyses published between 2009 and 2018 were identified in MEDLINE via PubMed and the Cochrane Database of Systematic Reviews. Data analysis was conducted from October 2019 to July 2024. Study selection: Meta-analyses of pharmacological interventions were eligible for inclusion if both randomized and nonrandomized studies contributed to a single meta-analytic estimate. Data extraction and synthesis: For this meta-analysis using a meta-epidemiological framework, separate summary effect size estimates were calculated for nonrandomized and randomized studies within each meta-analysis using a random-effects model and then these estimates were compared. The reporting of this study followed the Guidelines for Reporting Meta-Epidemiological Methodology Research and relevant portions of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guideline. Main outcome and measures: The primary outcome was discrepancies in treatment effects obtained from nonrandomized and randomized studies, as measured by the proportion of meta-analyses where the 2 study types disagreed about the direction or magnitude of effect, disagreed beyond chance about the effect size estimate, and the summary ratio of odds ratios (ROR) obtained from nonrandomized vs randomized studies combined across all meta-analyses. Results: A total of 346 meta-analyses with 2746 studies were included. Statistical conclusions about drug benefits and harms were different for 130 of 346 meta-analyses (37.6%) when focusing solely on either nonrandomized or randomized studies. Disagreements were beyond chance for 54 meta-analyses (15.6%). Across all meta-analyses, there was no strong evidence of consistent differences in treatment effects obtained from nonrandomized vs randomized studies (summary ROR, 0.95; 95% credible interval [CrI], 0.89-1.02). Compared with experimental nonrandomized studies, randomized studies produced on average a 19% smaller treatment effect (ROR, 0.81; 95% CrI, 0.68-0.97). There was increased heterogeneity in effect size estimates obtained from nonrandomized compared with randomized studies. Conclusions and relevance: In this meta-analysis of treatment effects of pharmacological interventions obtained from randomized and nonrandomized studies, there was no overall difference in effect size estimates between study types on average, but nonrandomized studies both overestimated and underestimated treatment effects observed in randomized studies and introduced additional uncertainty. These findings suggest that relying on nonrandomized studies as substitutes for RCTs may introduce additional uncertainty about the therapeutic effects of new drugs.

Interrupted Time-Series Analysis Yielded an Effect Estimate Concordant with the Cluster-Randomized Controlled Trial Result.

A Reanalysis of Cluster Randomized Trials Showed Interrupted Time-Series Studies Were Valuable in Health System Evaluation.

Methodology and reporting characteristics of studies using interrupted time series design in healthcare

Intent-to-treat Analysis of Cluster Randomized Trials when Clusters Report Unidentifiable Outcome Proportions.

Assessing Health Care Interventions via an Interrupted Time Series Model: Study Power and Design Considerations

Treatment Effects in RCTs and Nonrandomized Studies of Pharmacological Interventions

Use of a Statistical Adaptive Treatment Strategy Approach for Emulating Randomized Controlled Trials Using Observational Data: The Example of Blood-Pressure Control Strategies for the Prevention of Cardiovascular Events Among Individuals With Hypertension at High Cardiovascular Risk

INTERRUPTED TIME SERIES DESIGNS IN HEALTH TECHNOLOGY ASSESSMENT: LESSONS FROM TWO SYSTEMATIC REVIEWS OF BEHAVIOR CHANGE STRATEGIES

Small Studies May Overestimate the Effect Sizes in Critical Care Meta-Analyses: A Meta-Epidemiological Study

SECRETS: Subject-Efficient Clinical Randomized Controlled Trials using Synthetic Intervention

Design and statistical analysis reporting among interrupted time series studies in drug utilization research: a cross-sectional survey

A novel statistical method for comparing effectiveness of two treatments-simulated randomized controlled trials

A Practical Analysis Procedure on Generalizing Comparative Effectiveness in the Randomized Clinical Trial to the Real-world Trialeligible Population

Treatment Effects in Randomized and Nonrandomized Studies of Pharmacological Interventions: A Meta-Analysis

Estimating treatment effect in randomized trial after control to treatment crossover using external controls

Estimating an adjusted risk difference in a cluster randomized trial with individual-level analyses

A Robust Interrupted Time Series Model for Analyzing Complex Healthcare Intervention Data

Identification and estimation of treatment effects on long-term outcomes in clinical trials with external observational data

Blurring cluster randomized trials and observational studies: Two-Stage TMLE for subsampling, missingness, and few independent units.

Meta-analysis of community-based cluster randomization trials with binary outcomes

Series 2-19-2009 Multiple Imputation Methods for Treatment Noncompliance and Nonresponse in Randomized Clinical Trials