Abstract:Background: Policy evaluation studies that assess how state-level policies affect health-related outcomes are foundational to health and social policy research. The relative ability of newer analytic methods to address confounding, a key source of bias in observational studies, has not been closely examined. Methods: We conducted a simulation study to examine how differing magnitudes of confounding affected the performance of four methods used for policy evaluations: (1) the two-way fixed effects (TWFE) difference-in-differences (DID) model; (2) a one-period lagged autoregressive (AR) model; (3) augmented synthetic control method (ASCM); and (4) the doubly robust DID approach with multiple time periods from Callaway-Sant'Anna (CSA). We simulated our data to have staggered policy adoption and multiple confounding scenarios (i.e., varying the magnitude and nature of confounding relationships). Results: Bias increased for each method: (1) as confounding magnitude increases; (2) when confounding is generated with respect to prior outcome trends (rather than levels), and (3) when confounding associations are nonlinear (rather than linear). The AR and ASCM have notably lower root mean squared error than the TWFE model and CSA approach for all scenarios; the exception is nonlinear confounding by prior trends, where CSA excels. Coverage rates are unreasonably high for ASCM (e.g., 100%), reflecting large model-based standard errors and wide confidence intervals in practice. Conclusions: Our simulation study indicated that no single method consistently outperforms the others. But a researcher's toolkit should include all methodological options. Our simulations and associated R package can help researchers choose the most appropriate approach for their data.

Externally Valid Policy Evaluation Combining Trial and Observational Data

Selecting Experimental Sites for External Validity

Off-Policy Evaluation with Out-of-Sample Guarantees

A Trial Emulation Approach for Policy Evaluations with Group-level Longitudinal Data

Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation

Robust integration of external control data in randomized trials

Target Trial Emulation for Evaluating Health Policy

Using public clinical trial reports to evaluate observational study methods

Reference-trial-informed design to explore treatment effects in trial-underrepresented subgroups

Integrating Randomized Placebo-Controlled Trial Data with External Controls: A Semiparametric Approach with Selective Borrowing

Spatially Randomized Designs Can Enhance Policy Evaluation

Randomized trials and their observational emulations: a framework for benchmarking and joint analysis

Policy Evaluation for Temporal and/or Spatial Dependent Experiments

Combining Experimental and Historical Data for Policy Evaluation

Randomization Bias in Field Trials to Evaluate Targeting Methods

External Validity and Policy Adaptation: From Impact Evaluation to Policy Design

Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation

Identifying Optimal Methods for Addressing Confounding Bias When Estimating the Effects of State-Level Policies

Using Auxiliary Data to Guide the Recruitment of Sites for Randomized Controlled Trials

A Two-Stage Method for Extending Inferences from a Collection of Trials

Estimation and Inference on Treatment Effects Under Treatment-Based Sampling