Abstract:Model diagnostics and forecast evaluation are two sides of the same coin. A common principle is that fitted or predicted distributions ought to be calibrated or reliable, ideally in the sense of auto-calibration, where the outcome is a random draw from the posited distribution. For binary responses, this is the universal concept of reliability. For real-valued outcomes, a general theory of calibration has been elusive, despite a recent surge of interest in distributional regression and machine learning. We develop a framework rooted in probability theory, which gives rise to hierarchies of calibration, and applies to both predictive distributions and stand-alone point forecasts. In a nutshell, a prediction - distributional or single-valued - is conditionally T-calibrated if it can be taken at face value in terms of the functional T. Whenever T is defined via an identification function - as in the cases of threshold (non) exceedance probabilities, quantiles, expectiles, and moments - auto-calibration implies T-calibration. We introduce population versions of T-reliability diagrams and revisit a score decomposition into measures of miscalibration (MCB), discrimination (DSC), and uncertainty (UNC). In empirical settings, stable and efficient estimators of T-reliability diagrams and score components arise via nonparametric isotonic regression and the pool-adjacent-violators algorithm. For in-sample model diagnostics, we propose a universal coefficient of determination, $$\text{R}^\ast = \frac{\text{DSC}-\text{MCB}}{\text{UNC}},$$ that nests and reinterprets the classical $\text{R}^2$ in least squares (mean) regression and its natural analogue $\text{R}^1$ in quantile regression, yet applies to T-regression in general, with MCB $\geq 0$, DSC $\geq 0$, and $\text{R}^\ast \in [0,1]$ under modest conditions.

Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination

Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited

Residuals and Regression Diagnostics: Focusing on Logistic Regression

Decision Curve Analysis: a Technical Note

Calibration by Distribution Matching: Trainable Kernel Calibration Metrics

Uniform reliability tests for forecasting systems with small lead time

Diagnostic Checking for the Adequacy of Nonlinear Time Series Models

Calibrate: Interactive Analysis of Probabilistic Model Output

Testing For Auto-Calibration With Lorenz And Concentration Curves

Four Facets of Forecast Felicity: Calibration, Predictiveness, Randomness and Regret

Parity Calibration

U-Calibration: Forecasting for an Unknown Agent

Model diagnostics of discrete data regression: a unifying framework using functional residuals

Beyond the Norms: Detecting Prediction Errors in Regression Models

Understanding Reliability from a Regression Perspective

Scoring Rules and Calibration for Imprecise Probabilities

Probabilistic Scores of Classifiers, Calibration is not Enough

Auto-Calibration Tests for Discrete Finite Regression Functions

Reassessing How to Compare and Improve the Calibration of Machine Learning Models

Predictability Analysis of Regression Problems via Conditional Entropy Estimations

A Large-Scale Study of Probabilistic Calibration in Neural Network Regression