Abstract:Recent years have witnessed the success of introducing deep learning models to time series forecasting. From a data generation perspective, we illustrate that existing models are susceptible to distribution shifts driven by temporal contexts, whether observed or unobserved. Such context-driven distribution shift (CDS) introduces biases in predictions within specific contexts and poses challenges for conventional training paradigms. In this paper, we introduce a universal calibration methodology for the detection and adaptation of CDS with a trained model. To this end, we propose a novel CDS detector, termed the "residual-based CDS detector" or "Reconditionor", which quantifies the model's vulnerability to CDS by evaluating the mutual information between prediction residuals and their corresponding contexts. A high Reconditionor score indicates a severe susceptibility, thereby necessitating model adaptation. In this circumstance, we put forth a straightforward yet potent adapter framework for model calibration, termed the "sample-level contextualized adapter" or "SOLID". This framework involves the curation of a contextually similar dataset to the provided test sample and the subsequent fine-tuning of the model's prediction layer with a limited number of steps. Our theoretical analysis demonstrates that this adaptation strategy can achieve an optimal bias-variance trade-off. Notably, our proposed Reconditionor and SOLID are model-agnostic and readily adaptable to a wide range of models. Extensive experiments show that SOLID consistently enhances the performance of current forecasting models on real-world datasets, especially on cases with substantial CDS detected by the proposed Reconditionor, thus validating the effectiveness of the calibration approach.

Robust Calibration For Improved Weather Prediction Under Distributional Shift

Robust Calibration with Multi-domain Temperature Scaling

Deep Distributionally Robust Learning for Calibrated Uncertainties under Domain Shift

Learning Calibrated Uncertainties for Domain Shift: a Distributionally Robust Learning Approach

Post-hoc Uncertainty Calibration for Domain Drift Scenarios

Multiply Robust Estimation for Local Distribution Shifts with Multiple Domains

Robust Validation: Confident Predictions Even When Distributions Shift

Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift

Robust Conformal Prediction under Distribution Shift via Physics-Informed Structural Causal Model

A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift

Doubly robust calibration of prediction sets under covariate shift

Calibration of Time-Series Forecasting: Detecting and Adapting Context-Driven Distribution Shift

Model calibration via distributionally robust optimization: On the NASA Langley Uncertainty Quantification Challenge

Calibrating Bayesian UNet++ for Sub-Seasonal Forecasting

Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift

Quantifying Distribution Shifts and Uncertainties for Enhanced Model Robustness in Machine Learning Applications

t-RAIN: Robust generalization under weather-aliasing label shift attacks

Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibration

Confidence Calibration for Domain Generalization under Covariate Shift

Learning Optimal Classification Trees Robust to Distribution Shifts

Calibrated Probabilistic Forecasts for Arbitrary Sequences