Abstract:Conformal Predictive Systems (CPS) offer a versatile framework for constructing predictive distributions, allowing for calibrated inference and informative decision-making. However, their applicability has been limited to scenarios adhering to the Independent and Identically Distributed (IID) model assumption. This paper extends CPS to accommodate scenarios characterized by covariate shifts. We therefore propose Weighted CPS (WCPS), akin to Weighted Conformal Prediction (WCP), leveraging likelihood ratios between training and testing covariate distributions. This extension enables the construction of nonparametric predictive distributions capable of handling covariate shifts. We present theoretical underpinnings and conjectures regarding the validity and efficacy of WCPS and demonstrate its utility through empirical evaluations on both synthetic and real-world datasets. Our simulation experiments indicate that WCPS are probabilistically calibrated under covariate shift.
What problem does this paper attempt to address?
### Problems Addressed by the Paper
The paper aims to address the issue of covariate shift in predictive modeling. Specifically, existing Conformal Predictive Systems (CPS) perform poorly when dealing with inconsistent data distributions, i.e., when the covariate distributions of the training data and test data differ. This inconsistency is very common in practical applications, especially in fields such as time series analysis, counterfactual reasoning, and machine learning for scientific discovery.
To tackle this challenge, the authors propose Weighted Conformal Predictive Systems (WCPS), which construct calibrated predictive distributions by leveraging the likelihood ratio between the covariate distributions of the training data and test data. This approach enables CPS to maintain effective predictive performance even under covariate shift.
### Main Contributions
1. **Theoretical Framework**: The paper provides a detailed introduction to the theoretical foundation of WCPS, including its formal definition and properties, and proposes hypotheses regarding the probabilistic calibration of WCPS under covariate shift.
2. **Empirical Validation**: The effectiveness and probabilistic calibration of WCPS are validated through experiments on synthetic and real datasets.
3. **Extended Applications**: The potential applications of WCPS in counterfactual reasoning, label shift, feedback covariate shift, and survival analysis are discussed.
### Experimental Results
- **Coverage**: Under covariate shift, the prediction intervals generated by WCPS achieved the expected coverage level, whereas traditional CPS exhibited significant under-coverage.
- **Quality of Predictive Distributions**: Under covariate shift, the quality of the predictive distributions generated by WCPS (measured by Continuous Ranked Probability Score, CRPS) was superior to that of traditional CPS.
- **Probabilistic Calibration**: Simulation experiments demonstrated that the p-values generated by WCPS under covariate shift followed a uniform distribution, thereby proving its probabilistic calibration.
### Conclusion
The paper proposes a novel method—Weighted Conformal Predictive Systems (WCPS)—to address the issue of covariate shift in predictive modeling. Through theoretical analysis and empirical validation, WCPS is shown to be effective and reliable in handling covariate shift, providing strong support for practical applications. Future work will further refine the theoretical foundation and explore more application scenarios.