Abstract:We present an online post-hoc calibration method, called Online Platt Scaling (OPS), which combines the Platt scaling technique with online logistic regression. We demonstrate that OPS smoothly adapts between i.i.d. and non-i.i.d. settings with distribution drift. Further, in scenarios where the best Platt scaling model is itself miscalibrated, we enhance OPS by incorporating a recently developed technique called calibeating to make it more robust. Theoretically, our resulting OPS+calibeating method is guaranteed to be calibrated for adversarial outcome sequences. Empirically, it is effective on a range of synthetic and real-world datasets, with and without distribution drifts, achieving superior performance without hyperparameter tuning. Finally, we extend all OPS ideas to the beta scaling method.

What problem does this paper attempt to address?

### Problems the paper attempts to solve This paper aims to solve the calibration problem of machine - learning classifiers when facing data with distribution drift. Specifically, the paper focuses on the following two main problems: 1. **Can Platt Scaling (PS) be extended to handle data with distribution drift or change?** - The traditional Platt Scaling method assumes that the training and test data are independently and identically distributed (i.i.d.), but in practical applications, the data distribution may change over time (i.e., non - i.i.d. data). The paper proposes an online version of Platt Scaling (Online Platt Scaling, OPS), which can adapt to this distribution drift. 2. **How to use informative covariates (features) to achieve online adversarial calibration?** - Online adversarial calibration refers to achieving calibration by randomizing multiple probability predictions without relying on pre - trained models. The paper combines the recently proposed calibeating technique to improve the effect of online calibration and ensure its effectiveness in adversarial scenarios. ### Main methods and techniques - **Online Platt Scaling (OPS)**: Combine the traditional Platt Scaling method with online logistic regression, so that the model can continuously update parameters in the data stream, thus adapting to distribution drift. - **Calibeating**: Further apply the calibeating technique on the basis of OPS. By making corrections based on expert predictions, the calibration performance of the model is improved. Specifically, it includes two forms: - **Tracking**: Adjust predictions according to the average results in the same past intervals. - **Hedging**: Use a randomization strategy in each interval to ensure good calibration performance even in adversarial scenarios. ### Experimental verification The paper verifies the effectiveness of the proposed methods through a series of experiments, including performance on synthetic datasets and real - world datasets. The results show that OPS and OPS + calibeating can achieve better calibration effects and higher accuracy in various situations, especially in the presence of distribution drift. ### Summary This paper proposes a new online post - processing calibration method, Online Platt Scaling (OPS), and further enhances its robustness and adaptability by introducing the calibeating technique. These methods are not only theoretically guaranteed but also perform well in practical applications, providing new ideas for solving the calibration problem of machine - learning classifiers in dynamic environments.

Online Platt Scaling with Calibeating

Using Platt's scaling for calibration after undersampling -- limitations and how to address them

Online Calibrated and Conformal Prediction Improves Bayesian Optimization

Calibrated Regression Against An Adversary Without Regret

Adaptive Calibrator Ensemble for Model Calibration under Distribution Shift

Robust Calibration of Large Vision-Language Adapters

Scaling of Class-wise Training Losses for Post-hoc Calibration

On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines

Scaling up Data Augmentation MCMC via Calibration

Adaptive Training Distributions with Scalable Online Bilevel Optimization

Testing Calibration in Nearly-Linear Time

Bridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shift

Calibrated Computation-Aware Gaussian Processes

On Computationally Efficient Multi-Class Calibration

A Large-Scale Study of Probabilistic Calibration in Neural Network Regression

Gradient Methods with Online Scaling

Adaptive temperature scaling for Robust calibration of deep neural networks

Bayesian Adaptive Calibration and Optimal Design

Scaling Exponents Across Parameterizations and Optimizers

Optimizing Estimators of Squared Calibration Errors in Classification

A Self-boosted Framework for Calibrated Ranking