Abstract:Applications such as weather forecasting and personalized medicine demand models that output calibrated probability estimates---those representative of the true likelihood of a prediction. Most models are not calibrated out of the box but are recalibrated by post-processing model outputs. We find in this work that popular recalibration methods like Platt scaling and temperature scaling are (i) less calibrated than reported, and (ii) current techniques cannot estimate how miscalibrated they are. An alternative method, histogram binning, has measurable calibration error but is sample inefficient---it requires $O(B/\epsilon^2)$ samples, compared to $O(1/\epsilon^2)$ for scaling methods, where $B$ is the number of distinct probabilities the model can output. To get the best of both worlds, we introduce the scaling-binning calibrator, which first fits a parametric function to reduce variance and then bins the function values to actually ensure calibration. This requires only $O(1/\epsilon^2 + B)$ samples. Next, we show that we can estimate a model's calibration error more accurately using an estimator from the meteorological community---or equivalently measure its calibration error with fewer samples ($O(\sqrt{B})$ instead of $O(B)$). We validate our approach with multiclass calibration experiments on CIFAR-10 and ImageNet, where we obtain a 35% lower calibration error than histogram binning and, unlike scaling methods, guarantees on true calibration. In these experiments, we also estimate the calibration error and ECE more accurately than the commonly used plugin estimators. We implement all these methods in a Python library: <a class="link-external link-https" href="https://pypi.org/project/uncertainty-calibration" rel="external noopener nofollow">this https URL</a>

Calibrating Where It Matters: Constrained Temperature Scaling

On Temperature Scaling and Conformal Prediction of Deep Classifiers

Adaptive temperature scaling for Robust calibration of deep neural networks

On the Limitations of Temperature Scaling for Distributions with Overlaps

Delving into Temperature Scaling for Adaptive Conformal Prediction

Local temperature scaling for probability calibration

Does confidence calibration improve conformal prediction?

Learn then test: Calibrating predictive algorithms to achieve risk control

T-Cal: An optimal test for the calibration of predictive models

Unsupervised Temperature Scaling: An Unsupervised Post-Processing Calibration Method of Deep Networks

Dual-Branch Temperature Scaling Calibration for Long-Tailed Recognition

ESD: Expected Squared Difference as a Tuning-Free Trainable Calibration Measure

Improving Calibration by Relating Focal Loss, Temperature Scaling, and Properness

Calibration Error for Decision Making

From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration

Verified Uncertainty Calibration

Domain-adaptive and Subgroup-specific Cascaded Temperature Regression for Out-of-distribution Calibration

Cautious Calibration in Binary Classification

Optimizing Estimators of Squared Calibration Errors in Classification