Abstract:Randomized experiments are considered the gold standard for causal inference, as they can provide unbiased estimates of treatment effects for the experimental participants. However, researchers and policymakers are often interested in using a specific experiment to inform decisions about other target populations. In education research, increasing attention is being paid to the potential lack of generalizability of randomized experiments, as the experimental participants may be unrepresentative of the target population of interest. This paper examines whether generalization may be assisted by statistical methods that adjust for observed differences between the experimental participants and members of a target population. The methods examined include approaches that reweight the experimental data so that participants more closely resemble the target population and methods that utilize models of the outcome. Two simulation studies and one empirical analysis investigate and compare the methods' performance. One simulation uses purely simulated data while the other utilizes data from an evaluation of a school-based dropout prevention program. Our simulations suggest that machine learning methods outperform regression-based methods when the required structural (ignorability) assumptions are satisfied. When these assumptions are violated, all of the methods examined perform poorly. Our empirical analysis uses data from a multi-site experiment to assess how well results from a given site predict impacts in other sites. Using a variety of extrapolation methods, predicted effects for each site are compared to actual benchmarks. Flexible modeling approaches perform best, although linear regression is not far behind. Taken together, these results suggest that flexible modeling techniques can aid generalization while underscoring the fact that even state-of-the-art statistical techniques still rely on strong assumptions.

The Generalized Roy Model and the Cost-Benefit Analysis of Social Programs

Evaluating Marginal Policy Changes and the Average Effect of Treatment for Individuals at the Margin

Heterogeneous Treatment Effects in the Presence of Self-Selection: A Propensity Score Perspective

The Nonmarket Benefits of Education and Ability

Identifying the effects of a program offer with an application to Head Start

Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations

A Unified Framework for Efficient Estimation of General Treatment Models

Role models and revealed gender-specific costs of STEM in an extended Roy model of major choice

Sharp Bounds and Testability of a Roy Model of STEM Major Choices

Estimating Causal Moderation Effects with Randomized Treatments and Non-Randomized Moderators

Propensity Score–based Methods Versus MTE-based Methods in Causal Inference

The Generalized Risk-Adjusted Cost-Effectiveness (GRACE) Model for Measuring the Value of Gains in Health: An Exact Formulation

Net benefit separation and the determination curve: A probabilistic framework for cost-effectiveness estimation

Treatment Effects in Market Equilibrium

Estimating Heterogeneous Treatment Effects in the Presence of Self-Selection : A Propensity Score Perspective *

Empirical Welfare Maximization with Constraints

Incorporating Social Welfare in Program-Evaluation and Treatment Choice

Estimating treatment effect heterogeneity in randomized program evaluation

Exogenous Consideration and Extended Random Utility

Estimating Average Treatment Effects: Supplementary Analyses and Remaining Challenges

Estimating Welfare Effects in a Nonparametric Choice Model: The Case of School Vouchers