Abstract:Applications such as weather forecasting and personalized medicine demand models that output calibrated probability estimates---those representative of the true likelihood of a prediction. Most models are not calibrated out of the box but are recalibrated by post-processing model outputs. We find in this work that popular recalibration methods like Platt scaling and temperature scaling are (i) less calibrated than reported, and (ii) current techniques cannot estimate how miscalibrated they are. An alternative method, histogram binning, has measurable calibration error but is sample inefficient---it requires $O(B/\epsilon^2)$ samples, compared to $O(1/\epsilon^2)$ for scaling methods, where $B$ is the number of distinct probabilities the model can output. To get the best of both worlds, we introduce the scaling-binning calibrator, which first fits a parametric function to reduce variance and then bins the function values to actually ensure calibration. This requires only $O(1/\epsilon^2 + B)$ samples. Next, we show that we can estimate a model's calibration error more accurately using an estimator from the meteorological community---or equivalently measure its calibration error with fewer samples ($O(\sqrt{B})$ instead of $O(B)$). We validate our approach with multiclass calibration experiments on CIFAR-10 and ImageNet, where we obtain a 35% lower calibration error than histogram binning and, unlike scaling methods, guarantees on true calibration. In these experiments, we also estimate the calibration error and ECE more accurately than the commonly used plugin estimators. We implement all these methods in a Python library: <a class="link-external link-https" href="https://pypi.org/project/uncertainty-calibration" rel="external noopener nofollow">this https URL</a>

Computer model calibration with confidence and consistency

Efficient Calibration for Imperfect Computer Models

Bayesian Projected Calibration of Computer Models

Sobolev Calibration of Imperfect Computer Models

A review on computer model calibration

Can I Trust You? Rethinking Calibration with Controllable Confidence Ranking

Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates

Verified Uncertainty Calibration

Novel Model Calibration Method Via Non-Probabilistic Interval Characterization and Bayesian Theory

A Sequential Calibration and Validation Framework for Model Uncertainty Quantification and Reduction

Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration

Robust Experimental Designs for Model Calibration

Remarks On Parameter Calibration And Model Validation Based On Sensitivity

A Preposterior Analysis to Predict Identifiability in the Experimental Calibration of Computer Models

Nonparametric Functional Calibration of Computer Models

Efficiently Improving and Quantifying Robot Accuracy In Situ

Confidence Region for Calibration Function Coefficients

Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors

A Confidence Interval for the $\ell_2$ Expected Calibration Error

Model Validation and Calibration Based on Component Functions of Model Output.

A composite likelihood approach to computer model calibration using high-dimensional spatial data