Abstract:Model diagnostics and forecast evaluation are two sides of the same coin. A common principle is that fitted or predicted distributions ought to be calibrated or reliable, ideally in the sense of auto-calibration, where the outcome is a random draw from the posited distribution. For binary responses, this is the universal concept of reliability. For real-valued outcomes, a general theory of calibration has been elusive, despite a recent surge of interest in distributional regression and machine learning. We develop a framework rooted in probability theory, which gives rise to hierarchies of calibration, and applies to both predictive distributions and stand-alone point forecasts. In a nutshell, a prediction - distributional or single-valued - is conditionally T-calibrated if it can be taken at face value in terms of the functional T. Whenever T is defined via an identification function - as in the cases of threshold (non) exceedance probabilities, quantiles, expectiles, and moments - auto-calibration implies T-calibration. We introduce population versions of T-reliability diagrams and revisit a score decomposition into measures of miscalibration (MCB), discrimination (DSC), and uncertainty (UNC). In empirical settings, stable and efficient estimators of T-reliability diagrams and score components arise via nonparametric isotonic regression and the pool-adjacent-violators algorithm. For in-sample model diagnostics, we propose a universal coefficient of determination, $$\text{R}^\ast = \frac{\text{DSC}-\text{MCB}}{\text{UNC}},$$ that nests and reinterprets the classical $\text{R}^2$ in least squares (mean) regression and its natural analogue $\text{R}^1$ in quantile regression, yet applies to T-regression in general, with MCB $\geq 0$, DSC $\geq 0$, and $\text{R}^\ast \in [0,1]$ under modest conditions.

Confidence Region for Calibration Function Coefficients

Calibration for the Errors Resulted from Aberration in Long Focal Length Measurement

Registration Error Analysis and Evaluation in Large-Volume Metrology System

Calibration of a Multi-Beam LIDAR by Using Linear Instrumental Error Model

Study on Calibration Method and System Error Compensation of Balancing Machines

Model Order Selection In Errors-In-Variables System Identification

Computer model calibration with confidence and consistency

A Confidence Interval for the $\ell_2$ Expected Calibration Error

Sensitivity and Uncertainty Analysis of Nonideal Calibration Standards for 16-Term Error Model

Cubature-based uncertainty estimation for nonlinear regression models

Force calibration using errors-in-variables regression and Monte Carlo uncertainty evaluation

Efficient Calibration for Imperfect Computer Models

Homoscedastic controlled calibration model

Verified Uncertainty Calibration

Accurate Uncertainties for Deep Learning Using Calibrated Regression

Consistent and Asymptotically Unbiased Estimation of Proper Calibration Errors

Evaluating and Calibrating Uncertainty Prediction in Regression Tasks

Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression tasks

Expectation consistency for calibration of neural networks

Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination

Assessment Of Error Assumption In Probabilistic Model Calibration Of Rainfall Infiltration In Soil Slope