Sensitivity analysis for the generalization of experimental results

Melody Y Huang
DOI: https://doi.org/10.1093/jrsssa/qnae012
2024-03-13
Abstract:Abstract Randomized controlled trials (RCT’s) allow researchers to estimate causal effects in an experimental sample with minimal identifying assumptions. However, to generalize or transport a causal effect from an RCT to a target population, researchers must adjust for a set of treatment effect moderators. In practice, it is impossible to know whether the set of moderators has been properly accounted for. I propose a two parameter sensitivity analysis for generalizing or transporting experimental results using weighted estimators. The contributions in the article are threefold. First, I show that the sensitivity parameters are scale-invariant and standardized, and introduce an estimation approach for researchers to account for both bias in their estimates from omitting a moderator, as well as potential changes to their inference. Second, I propose several tools researchers can use to perform sensitivity analysis: (1) numerical measures to summarize the uncertainty in an estimated effect to omitted moderators; (2) graphical summary tools to visualize the sensitivity in estimated effects; and (3) a formal benchmarking approach for researchers to estimate potential sensitivity parameter values using existing data. Finally, I demonstrate that the proposed framework can be easily extended to the class of doubly robust, augmented weighted estimators.
statistics & probability,social sciences, mathematical methods
What problem does this paper attempt to address?