Abstract:Most machine learning classifiers are designed to output posterior probabilities for the classes given the input sample. These probabilities may be used to make the categorical decision on the class of the sample; provided as input to a downstream system; or provided to a human for interpretation. Evaluating the quality of the posteriors generated by these system is an essential problem which was addressed decades ago with the invention of proper scoring rules (PSRs). Unfortunately, much of the recent machine learning literature uses calibration metrics -- most commonly, the expected calibration error (ECE) -- as a proxy to assess posterior performance. The problem with this approach is that calibration metrics reflect only one aspect of the quality of the posteriors, ignoring the discrimination performance. For this reason, we argue that calibration metrics should play no role in the assessment of posterior quality. Expected PSRs should instead be used for this job, preferably normalized for ease of interpretation. In this work, we first give a brief review of PSRs from a practical perspective, motivating their definition using Bayes decision theory. We discuss why expected PSRs provide a principled measure of the quality of a system's posteriors and why calibration metrics are not the right tool for this job. We argue that calibration metrics, while not useful for performance assessment, may be used as diagnostic tools during system development. With this purpose in mind, we discuss a simple and practical calibration metric, called calibration loss, derived from a decomposition of expected PSRs. We compare this metric with the ECE and with the expected score divergence calibration metric from the PSR literature and argue, using theoretical and empirical evidence, that calibration loss is superior to these two metrics.

Obtaining Calibrated Probabilities from Boosting

Probabilistic Scores of Classifiers, Calibration is not Enough

Calibrated Boosting-Forest

Cautious Calibration in Binary Classification

Calibration of Machine Learning Classifiers for Probability of Default Modelling

Optimizing Calibration by Gaining Aware of Prediction Correctness

From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration

A Large-Scale Study of Probabilistic Calibration in Neural Network Regression

Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration

Improving Inverse Probability Weighting by Post-calibrating Its Propensity Scores

Calibrate: Interactive Analysis of Probabilistic Model Output

Experiments with a New Boosting Algorithm

Risk-based Calibration for Probabilistic Classifiers

Calibrating sufficiently

Risk prediction models for discrete ordinal outcomes: calibration and the impact of the proportional odds assumption

Probability Estimation for Multi-Class Classification Using AdaBoost

When Analytic Calculus Cracks AdaBoost Code

When Does Optimizing a Proper Loss Yield Calibration?

Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions

Confidence Calibration of Classifiers with Many Classes

Calibration Error for Decision Making