Abstract:Applications such as weather forecasting and personalized medicine demand models that output calibrated probability estimates---those representative of the true likelihood of a prediction. Most models are not calibrated out of the box but are recalibrated by post-processing model outputs. We find in this work that popular recalibration methods like Platt scaling and temperature scaling are (i) less calibrated than reported, and (ii) current techniques cannot estimate how miscalibrated they are. An alternative method, histogram binning, has measurable calibration error but is sample inefficient---it requires $O(B/\epsilon^2)$ samples, compared to $O(1/\epsilon^2)$ for scaling methods, where $B$ is the number of distinct probabilities the model can output. To get the best of both worlds, we introduce the scaling-binning calibrator, which first fits a parametric function to reduce variance and then bins the function values to actually ensure calibration. This requires only $O(1/\epsilon^2 + B)$ samples. Next, we show that we can estimate a model's calibration error more accurately using an estimator from the meteorological community---or equivalently measure its calibration error with fewer samples ($O(\sqrt{B})$ instead of $O(B)$). We validate our approach with multiclass calibration experiments on CIFAR-10 and ImageNet, where we obtain a 35% lower calibration error than histogram binning and, unlike scaling methods, guarantees on true calibration. In these experiments, we also estimate the calibration error and ECE more accurately than the commonly used plugin estimators. We implement all these methods in a Python library: <a class="link-external link-https" href="https://pypi.org/project/uncertainty-calibration" rel="external noopener nofollow">this https URL</a>

Evidential calibration of binary SVM classifiers

On the Calibration of Probabilistic Classifier Sets

From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration

Classifier Calibration: with application to threat scores in cybersecurity

Classifier Calibration: A survey on how to assess and improve predicted class probabilities

Probabilistic Scores of Classifiers, Calibration is not Enough

Calibration methods in imbalanced binary classification

Quantifying calibration error in modern neural networks through evidence based theory

Cautious Calibration in Binary Classification

Verified Uncertainty Calibration

Confidence Calibration of Classifiers with Many Classes

Sparse Bayesian Mixture of Experts and Its Application to Spectral Multivariate Calibration

Calibrating Expressions of Certainty

Evidential Uncertainty Quantification: A Variance-Based Perspective

Optimizing Estimators of Squared Calibration Errors in Classification

Multi-Class Svm Classifiers Fusion Based On Evidence Combination

Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates

Evidence combination with multi-granularity belief structure for pattern classification

Calibrating Verbalized Probabilities for Large Language Models

Multi-sensor data fusion based on the belief divergence measure of evidences and the belief entropy

BSM loss: A superior way in modeling aleatory uncertainty of fine_grained classification