Abstract:Applications such as weather forecasting and personalized medicine demand models that output calibrated probability estimates---those representative of the true likelihood of a prediction. Most models are not calibrated out of the box but are recalibrated by post-processing model outputs. We find in this work that popular recalibration methods like Platt scaling and temperature scaling are (i) less calibrated than reported, and (ii) current techniques cannot estimate how miscalibrated they are. An alternative method, histogram binning, has measurable calibration error but is sample inefficient---it requires $O(B/\epsilon^2)$ samples, compared to $O(1/\epsilon^2)$ for scaling methods, where $B$ is the number of distinct probabilities the model can output. To get the best of both worlds, we introduce the scaling-binning calibrator, which first fits a parametric function to reduce variance and then bins the function values to actually ensure calibration. This requires only $O(1/\epsilon^2 + B)$ samples. Next, we show that we can estimate a model's calibration error more accurately using an estimator from the meteorological community---or equivalently measure its calibration error with fewer samples ($O(\sqrt{B})$ instead of $O(B)$). We validate our approach with multiclass calibration experiments on CIFAR-10 and ImageNet, where we obtain a 35% lower calibration error than histogram binning and, unlike scaling methods, guarantees on true calibration. In these experiments, we also estimate the calibration error and ECE more accurately than the commonly used plugin estimators. We implement all these methods in a Python library: <a class="link-external link-https" href="https://pypi.org/project/uncertainty-calibration" rel="external noopener nofollow">this https URL</a>

A Confidence Interval for the $\ell_2$ Expected Calibration Error

Calibration Error for Decision Making

Expectation consistency for calibration of neural networks

Information-theoretic Generalization Analysis for Expected Calibration Error

Can I Trust You? Rethinking Calibration with Controllable Confidence Ranking

T-Cal: An optimal test for the calibration of predictive models

Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration

Beyond Calibration: Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence

Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression tasks

Verified Uncertainty Calibration

Accurate Uncertainties for Deep Learning Using Calibrated Regression

A Consistent and Differentiable Lp Canonical Calibration Error Estimator

A Unifying Theory of Distance from Calibration

ESD: Expected Squared Difference as a Tuning-Free Trainable Calibration Measure

Optimizing Estimators of Squared Calibration Errors in Classification

How Flawed Is ECE? An Analysis via Logit Smoothing

Consistent and Asymptotically Unbiased Estimation of Proper Calibration Errors

Calibrated Prediction Intervals for Neural Network Regressors

Rethinking Calibration of Deep Neural Networks: Do Not Be Afraid of Overconfidence

From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration

Evaluating and Calibrating Uncertainty Prediction in Regression Tasks