Abstract:Many existing approaches to generalizing statistical inference amidst distribution shift operate under the covariate shift assumption, which posits that the conditional distribution of unobserved variables given observable ones is invariant across populations. However, recent empirical investigations have demonstrated that adjusting for shift in observed variables (covariate shift) is often insufficient for generalization. In other words, covariate shift does not typically ``explain away'' the distribution shift between settings. As such, addressing the unknown yet non-negligible shift in the unobserved variables given observed ones (conditional shift) is crucial for generalizable inference. In this paper, we present a series of empirical evidence from two large-scale multi-site replication studies to support a new role of covariate shift in ``predicting'' the strength of the unknown conditional shift. Analyzing 680 studies across 65 sites, we find that even though the conditional shift is non-negligible, its strength can often be bounded by that of the observable covariate shift. However, this pattern only emerges when the two sources of shifts are quantified by our proposed standardized, ``pivotal'' measures. We then interpret this phenomenon by connecting it to similar patterns that can be theoretically derived from a random distribution shift model. Finally, we demonstrate that exploiting the predictive role of covariate shift leads to reliable and efficient uncertainty quantification for target estimates in generalization tasks with partially observed data. Overall, our empirical and theoretical analyses suggest a new way to approach the problem of distributional shift, generalizability, and external validity.

Rectifying Group Irregularities in Explanations for Distribution Shift

Rethinking Distribution Shifts: Empirical Analysis and Inductive Modeling for Tabular Data

Fairness Hub Technical Briefs: Definition and Detection of Distribution Shift

On Minimizing the Impact of Dataset Shifts on Actionable Explanations

Beyond Discrepancy: A Closer Look at the Theory of Distribution Shift

When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights

Decision-Focused Evaluation of Worst-Case Distribution Shift

Beyond Reweighting: On the Predictive Role of Covariate Shift in Effect Generalization

Change is Hard: A Closer Look at Subpopulation Shift

Enhancing Distributional Stability among Sub-populations

Why Groups Matter: Necessity of Group Structures in Attributions

View From Above: A Framework for Evaluating Distribution Shifts in Model Behavior

Are Data-driven Explanations Robust against Out-of-distribution Data?

Understanding Disparities in Post Hoc Machine Learning Explanation

Shift-Share Designs: Theory and Inference

Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts

Learning Groupwise Explanations for Black-Box Models.

Control+Shift: Generating Controllable Distribution Shifts

Handling Distribution Shifts on Graphs: An Invariance Perspective

Multiply Robust Estimation for Local Distribution Shifts with Multiple Domains

Interpretable Data-Based Explanations for Fairness Debugging