Abstract:As machine learning models are increasingly deployed in dynamic environments, it becomes paramount to assess and quantify uncertainties associated with distribution shifts. A distribution shift occurs when the underlying data-generating process changes, leading to a deviation in the model's performance. The prediction interval, which captures the range of likely outcomes for a given prediction, serves as a crucial tool for characterizing uncertainties induced by their underlying distribution. In this paper, we propose methodologies for aggregating prediction intervals to obtain one with minimal width and adequate coverage on the target domain under unsupervised domain shift, under which we have labeled samples from a related source domain and unlabeled covariates from the target domain. Our analysis encompasses scenarios where the source and the target domain are related via i) a bounded density ratio, and ii) a measure-preserving transformation. Our proposed methodologies are computationally efficient and easy to implement. Beyond illustrating the performance of our method through real-world datasets, we also delve into the theoretical details. This includes establishing rigorous theoretical guarantees, coupled with finite sample bounds, regarding the coverage and width of our prediction intervals. Our approach excels in practical applications and is underpinned by a solid theoretical framework, ensuring its reliability and effectiveness across diverse contexts.

Unsupervised Calibration under Covariate Shift

Confidence Calibration for Domain Generalization under Covariate Shift

Post-hoc Uncertainty Calibration for Domain Drift Scenarios

Doubly robust calibration of prediction sets under covariate shift

Transferable Calibration with Lower Bias and Variance in Domain Adaptation.

Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions

Learning Calibrated Uncertainties for Domain Shift: a Distributionally Robust Learning Approach

A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift

Leveraging Unsupervised Data and Domain Adaptation for Deep Regression in Low-Cost Sensor Calibration

Bridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shift

Privacy Preserving Recalibration under Domain Shift

Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift

A One-step Approach to Covariate Shift Adaptation

Estimating calibration error under label shift without labels

Learn then test: Calibrating predictive algorithms to achieve risk control

A Large-Scale Study of Probabilistic Calibration in Neural Network Regression

Calibration of Network Confidence for Unsupervised Domain Adaptation Using Estimated Accuracy

Robust Calibration with Multi-domain Temperature Scaling

Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibration

Unsupervised Domain Adaptation via Calibrating Uncertainties