Abstract:Many existing approaches to generalizing statistical inference amidst distribution shift operate under the covariate shift assumption, which posits that the conditional distribution of unobserved variables given observable ones is invariant across populations. However, recent empirical investigations have demonstrated that adjusting for shift in observed variables (covariate shift) is often insufficient for generalization. In other words, covariate shift does not typically ``explain away'' the distribution shift between settings. As such, addressing the unknown yet non-negligible shift in the unobserved variables given observed ones (conditional shift) is crucial for generalizable inference. In this paper, we present a series of empirical evidence from two large-scale multi-site replication studies to support a new role of covariate shift in ``predicting'' the strength of the unknown conditional shift. Analyzing 680 studies across 65 sites, we find that even though the conditional shift is non-negligible, its strength can often be bounded by that of the observable covariate shift. However, this pattern only emerges when the two sources of shifts are quantified by our proposed standardized, ``pivotal'' measures. We then interpret this phenomenon by connecting it to similar patterns that can be theoretically derived from a random distribution shift model. Finally, we demonstrate that exploiting the predictive role of covariate shift leads to reliable and efficient uncertainty quantification for target estimates in generalization tasks with partially observed data. Overall, our empirical and theoretical analyses suggest a new way to approach the problem of distributional shift, generalizability, and external validity.

Shift-Share Designs: Theory and Inference

Relaxing the exclusion restriction in shift-share instrumental variable estimation

Overidentification in Shift-Share Designs

Design-Based Inference for Spatial Experiments under Unknown Interference

Rethinking Distribution Shifts: Empirical Analysis and Inductive Modeling for Tabular Data

Inference in Spatial Experiments with Interference using the SpatialEffect Package

Rectifying Group Irregularities in Explanations for Distribution Shift

Beyond Reweighting: On the Predictive Role of Covariate Shift in Effect Generalization

Estimation and Inference for Synthetic Control Methods with Spillover Effects

Regression adjustments for estimating the global treatment effect in experiments with interference

A Unifying Causal Framework for Analyzing Dataset Shift-stable Learning Algorithms

Efficient and Multiply Robust Risk Estimation under General Forms of Dataset Shift

Distribution-Free Prediction Intervals Under Covariate Shift, With an Application to Causal Inference

Beyond Discrepancy: A Closer Look at the Theory of Distribution Shift

Robustness, Heterogeneous Treatment Effects and Covariate Shifts

Causal inference for interfering units with cluster and population level treatment allocation programs

A quantile shift approach to main effects and interactions in a 2-by-2 design

Identification and estimation of spillover effects in randomized experiments

Research on Regional Structural Changes and Spatial Distribution of Construction Land Based on Shift-Share Analysis

A Simple Model to Estimate Sharing Effects in Social Networks

Sequential Data Integration Under Dataset Shift