Abstract:Randomized experiments are considered the gold standard for causal inference, as they can provide unbiased estimates of treatment effects for the experimental participants. However, researchers and policymakers are often interested in using a specific experiment to inform decisions about other target populations. In education research, increasing attention is being paid to the potential lack of generalizability of randomized experiments, as the experimental participants may be unrepresentative of the target population of interest. This paper examines whether generalization may be assisted by statistical methods that adjust for observed differences between the experimental participants and members of a target population. The methods examined include approaches that reweight the experimental data so that participants more closely resemble the target population and methods that utilize models of the outcome. Two simulation studies and one empirical analysis investigate and compare the methods' performance. One simulation uses purely simulated data while the other utilizes data from an evaluation of a school-based dropout prevention program. Our simulations suggest that machine learning methods outperform regression-based methods when the required structural (ignorability) assumptions are satisfied. When these assumptions are violated, all of the methods examined perform poorly. Our empirical analysis uses data from a multi-site experiment to assess how well results from a given site predict impacts in other sites. Using a variety of extrapolation methods, predicted effects for each site are compared to actual benchmarks. Flexible modeling approaches perform best, although linear regression is not far behind. Taken together, these results suggest that flexible modeling techniques can aid generalization while underscoring the fact that even state-of-the-art statistical techniques still rely on strong assumptions.

Generalizing and transporting causal inferences from randomized trials in the presence of trial engagement effects

Generalizing and transporting inferences about the effects of treatment assignment subject to non-adherence

Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals

Towards causally interpretable meta-analysis: transporting inferences from multiple studies to a target population

Towards Generalizing Inferences from Trials to Target Populations

Study designs for extending causal inferences from a randomized trial to a target population

Causal inference under transportability assumptions for conditional relative effect measures

Learning about treatment effects in a new target population under transportability assumptions for relative effect measures

Estimating individual treatment effect: generalization bounds and algorithms

Prediction-powered Generalization of Causal Inferences

Estimating subgroup effects in generalizability and transportability analyses

Methods for Extending Inferences From Observational Studies: Considering Causal Structures, Identification Assumptions, and Estimators

Extending inferences from a randomized trial to a new target population

Regression-based estimation of heterogeneous treatment effects when extending inferences from a randomized trial to a target population

Improving transportability of randomized controlled trial inference using robust prediction methods

Efficient Generalization and Transportation

Extending inferences from a cluster randomized trial to a target population

Forecasting Causal Effects of Future Interventions: Confounding and Transportability Issues

Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations

Identifiability of Subgroup Causal Effects in Randomized Experiments with Nonignorable Missing Covariates

Generalizing treatment effects with incomplete covariates: Identifying assumptions and multiple imputation algorithms