Abstract:We introduce a fine-grained framework for uncertainty quantification of predictive models under distributional shifts. This framework distinguishes the shift in covariate distributions from that in the conditional relationship between the outcome ($Y$) and the covariates ($X$). We propose to reweight the training samples to adjust for an identifiable covariate shift while protecting against worst-case conditional distribution shift bounded in an $f$-divergence ball. Based on ideas from conformal inference and distributionally robust learning, we present an algorithm that outputs (approximately) valid and efficient prediction intervals in the presence of distributional shifts. As a use case, we apply the framework to sensitivity analysis of individual treatment effects with hidden confounding. The proposed methods are evaluated in simulation studies and three real data applications, demonstrating superior robustness and efficiency compared with existing benchmarks.
What problem does this paper attempt to address?
This paper aims to solve the problem of performance degradation of prediction models in the face of distributional shifts. Specifically, the authors introduce a fine - grained framework for quantifying the uncertainty of prediction models in the case of distributional shifts. This framework distinguishes between changes in the covariate distribution and changes in the conditional relationship between the outcome (Y) and the covariate (X). The main contributions of the paper include:
1. **Proposing Weighted Robust Conformal Prediction (WRCP)**: This method can handle covariate shift and conditional distribution shift. It re - weights the training samples to adjust the identifiable covariate shift and protects against the worst - case conditional distribution shift, ensuring that the prediction interval remains valid and efficient in the presence of distributional shifts.
2. **Debiased WRCP (D - WRCP)**: When the covariates are high - dimensional, it may be difficult to directly estimate the covariate shift. D - WRCP uses de - biasing techniques to construct effective prediction intervals, which has double robustness, and its mis - coverage rate depends on the product of the covariate likelihood ratio estimation error and the conditional quantile residual estimation error.
3. **Application to Sensitivity Analysis of Individual Treatment Effects (ITE)**: The paper shows that the proposed method can be adapted to perform sensitivity analysis of individual treatment effects under the f - sensitivity model, thereby evaluating the robustness of causal effect estimates in the presence of hidden confounding factors.
### Paper Background
In real - world applications, prediction models are often required to be deployed in environments with different distributions from the training data. This distribution difference may lead to a significant decline in model performance. Especially in high - risk scenarios, providing calibrated uncertainty quantification becomes particularly important. Although traditional Conformal Prediction (CP) methods can generate valid prediction intervals, their validity depends on the exchangeability assumption between training data and test data, which no longer holds in the presence of distributional shifts. For this reason, previous studies have proposed robust CP methods for the worst - case joint distribution shift, but these methods fail to distinguish between covariate shift and conditional distribution shift, resulting in potentially being too conservative in practical applications.
### Main Methods
The paper achieves fine - grained robust prediction inference through the following steps:
1. **Decomposing Distributional Shifts**: Divide distributional shifts into two categories:
- **Covariate Shift**: The marginal distributions of covariates in the training environment and the target environment are different.
- **Conditional Relationship Shift**: The conditional relationships between the outcome and the covariates in the training environment and the target environment are different.
2. **Constructing Prediction Intervals**: For a given covariate \( X_{n + 1} \), construct a prediction interval by re - weighting the training samples and adjusting the confidence level to ensure a high probability of covering the true outcome under the target distribution.
3. **Double Robustness**: When the covariates are high - dimensional, D - WRCP ensures the validity of the prediction interval by combining the estimates of covariate shift and conditional quantile residuals.
### Application Examples
The paper applies the proposed method to the sensitivity analysis of individual treatment effects, especially in the presence of hidden confounding factors. Through simulation studies and three real - data applications, the effectiveness and efficiency of the proposed method are verified.
### Conclusion
This paper proposes a fine - grained robust prediction inference framework that can provide effective uncertainty quantification in the presence of distributional shifts. By distinguishing between covariate shift and conditional distribution shift, this method exhibits higher efficiency and robustness in practical applications.