Identifying Optimal Methods for Addressing Confounding Bias When Estimating the Effects of State-Level Policies

Beth Ann Griffin,Megan S. Schuler,Elizabeth M. Stone,Stephen W. Patrick,Bradley D. Stein,Pedro Nascimento de Lima,Max Griswold,Adam Scherling,Elizabeth A. Stuart

DOI: https://doi.org/10.48550/arXiv.2307.01413

2023-07-04

Abstract:Background: Policy evaluation studies that assess how state-level policies affect health-related outcomes are foundational to health and social policy research. The relative ability of newer analytic methods to address confounding, a key source of bias in observational studies, has not been closely examined. Methods: We conducted a simulation study to examine how differing magnitudes of confounding affected the performance of four methods used for policy evaluations: (1) the two-way fixed effects (TWFE) difference-in-differences (DID) model; (2) a one-period lagged autoregressive (AR) model; (3) augmented synthetic control method (ASCM); and (4) the doubly robust DID approach with multiple time periods from Callaway-Sant'Anna (CSA). We simulated our data to have staggered policy adoption and multiple confounding scenarios (i.e., varying the magnitude and nature of confounding relationships). Results: Bias increased for each method: (1) as confounding magnitude increases; (2) when confounding is generated with respect to prior outcome trends (rather than levels), and (3) when confounding associations are nonlinear (rather than linear). The AR and ASCM have notably lower root mean squared error than the TWFE model and CSA approach for all scenarios; the exception is nonlinear confounding by prior trends, where CSA excels. Coverage rates are unreasonably high for ASCM (e.g., 100%), reflecting large model-based standard errors and wide confidence intervals in practice. Conclusions: Our simulation study indicated that no single method consistently outperforms the others. But a researcher's toolkit should include all methodological options. Our simulations and associated R package can help researchers choose the most appropriate approach for their data.

Methodology

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively deal with confounding bias when evaluating the impact of state - level policies on health - related outcomes. Specifically, the paper compares the performance of four commonly used analysis methods under different degrees of confounding through simulation studies. These four methods are:

1. **Two - Way Fixed Effects (TWFE) Difference - in - Differences (DID) model**: This is a commonly used method for evaluating policy effects. It estimates policy effects by comparing the changes in the treatment group and the control group before and after policy implementation.
2. **One - Period Lagged Autoregressive (AR) model**: This method reduces confounding by controlling for time - fixed effects, lagged outcome values, and other time - invariant or time - varying state - level confounding factors.
3. **Augmented Synthetic Control Method (ASCM)**: This method makes the treatment group and the control group comparable in each policy - implementation - year cohort through weighting and controls confounding through additional covariate adjustment.
4. **Doubly Robust DID Approach with Multiple Time Periods (CSA)**: This method aims to overcome the known biases of the classic TWFE model in the presence of heterogeneous treatment effects and phased implementation. It controls confounding through phased weighting and covariate adjustment.

The core problem of the paper is to evaluate the performance of these methods under different degrees of confounding, especially when confounding is caused by previous outcome trends rather than levels, and when the confounding relationship is nonlinear rather than linear. Through simulation studies, the author hopes to provide researchers with guidance on choosing the most appropriate method to more accurately evaluate the effects of state - level policies.