Abstract:Many existing approaches to generalizing statistical inference amidst distribution shift operate under the covariate shift assumption, which posits that the conditional distribution of unobserved variables given observable ones is invariant across populations. However, recent empirical investigations have demonstrated that adjusting for shift in observed variables (covariate shift) is often insufficient for generalization. In other words, covariate shift does not typically ``explain away'' the distribution shift between settings. As such, addressing the unknown yet non-negligible shift in the unobserved variables given observed ones (conditional shift) is crucial for generalizable inference. In this paper, we present a series of empirical evidence from two large-scale multi-site replication studies to support a new role of covariate shift in ``predicting'' the strength of the unknown conditional shift. Analyzing 680 studies across 65 sites, we find that even though the conditional shift is non-negligible, its strength can often be bounded by that of the observable covariate shift. However, this pattern only emerges when the two sources of shifts are quantified by our proposed standardized, ``pivotal'' measures. We then interpret this phenomenon by connecting it to similar patterns that can be theoretically derived from a random distribution shift model. Finally, we demonstrate that exploiting the predictive role of covariate shift leads to reliable and efficient uncertainty quantification for target estimates in generalization tasks with partially observed data. Overall, our empirical and theoretical analyses suggest a new way to approach the problem of distributional shift, generalizability, and external validity.

Estimating Confidence Interval of Variance Component for Skewed Distrubition Data in Generalizability Theory

Robust Estimation and Shrinkage in Ultrahigh Dimensional Expectile Regression with Heavy Tails and Variance Heterogeneity

A Monte Carlo Study of Confidence Interval Methods for Generalizability Coefficient.

Problematic standard errors and confidence intervals for skewness and kurtosis

Comparing Confidence Intervals for the Mean of Symmetric and Skewed Distributions

Generating Multivariate Ordinal Data via Entropy Principles

A method of generating multivariate non-normal random numbers with desired multivariate skewness and kurtosis

Confidence Corridors for Multivariate Generalized Quantile Regression

A Bayesian approach to estimating variance components within a multivariate generalizability theory framework

Univariate and multivariate skewness and kurtosis for measuring nonnormality: Prevalence, influence and estimation

The Effect of Skewness and Kurtosis on Mean and Covariance Structure Analysis

Calculation of confidence intervals for simulation output

Beyond Reweighting: On the Predictive Role of Covariate Shift in Effect Generalization

Robust inference for skewed data in health sciences

Estimating the Confidence Interval for the Common Coefficient of Variation for Multiple Inverse Gaussian Distributions

Growth Estimators and Confidence Intervals for the Mean of Negative Binomial Random Variables with Unknown Dispersion

Spline Confidence Bands for Variance Functions

Statistical Inference for Covariate-Adjusted and Interpretable Generalized Factor Model with Application to Testing Fairness

Inference on order restricted means of inverse Gaussian populations under heteroscedasticity

Family of Kumaraswamy generalized distribution in estimation of multi-component reliability under the adaptive hybrid progressive censoring schemes

Adjusted Confidence Bands in Nonparametric Regression