Abstract:Applications such as weather forecasting and personalized medicine demand models that output calibrated probability estimates---those representative of the true likelihood of a prediction. Most models are not calibrated out of the box but are recalibrated by post-processing model outputs. We find in this work that popular recalibration methods like Platt scaling and temperature scaling are (i) less calibrated than reported, and (ii) current techniques cannot estimate how miscalibrated they are. An alternative method, histogram binning, has measurable calibration error but is sample inefficient---it requires $O(B/\epsilon^2)$ samples, compared to $O(1/\epsilon^2)$ for scaling methods, where $B$ is the number of distinct probabilities the model can output. To get the best of both worlds, we introduce the scaling-binning calibrator, which first fits a parametric function to reduce variance and then bins the function values to actually ensure calibration. This requires only $O(1/\epsilon^2 + B)$ samples. Next, we show that we can estimate a model's calibration error more accurately using an estimator from the meteorological community---or equivalently measure its calibration error with fewer samples ($O(\sqrt{B})$ instead of $O(B)$). We validate our approach with multiclass calibration experiments on CIFAR-10 and ImageNet, where we obtain a 35% lower calibration error than histogram binning and, unlike scaling methods, guarantees on true calibration. In these experiments, we also estimate the calibration error and ECE more accurately than the commonly used plugin estimators. We implement all these methods in a Python library: <a class="link-external link-https" href="https://pypi.org/project/uncertainty-calibration" rel="external noopener nofollow">this https URL</a>

Calibrating general posterior credible regions

Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probability

Practical calibration of the temperature parameter in Gibbs posteriors

Calibrating Expressions of Certainty

Calibrating approximate Bayesian credible intervals of gravitational-wave parameters

Beyond Calibration: Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence

Cautious Calibration in Binary Classification

Confidence Intervals and Regions for Quantiles Using Conditional Monte Carlo and Generalized Likelihood Ratios.

Calibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inference

On the Calibration of Probabilistic Classifier Sets

From Conformal Predictions to Confidence Regions

Model-free posterior inference on the area under the receiver operating characteristic curve

Non‐parametric inference on calibration of predicted risks

Monte Carlo Confidence Sets for Identified Sets

Model calibration via distributionally robust optimization: On the NASA Langley Uncertainty Quantification Challenge

Adjusting Regression Models for Conditional Uncertainty Calibration

Computer model calibration with confidence and consistency

Bayesian Model Calibration for Extrapolative Prediction via Gibbs Posteriors

Calibrating Model-Based Inferences and Decisions

Verified Uncertainty Calibration