Abstract:Model diagnostics and forecast evaluation are two sides of the same coin. A common principle is that fitted or predicted distributions ought to be calibrated or reliable, ideally in the sense of auto-calibration, where the outcome is a random draw from the posited distribution. For binary responses, this is the universal concept of reliability. For real-valued outcomes, a general theory of calibration has been elusive, despite a recent surge of interest in distributional regression and machine learning. We develop a framework rooted in probability theory, which gives rise to hierarchies of calibration, and applies to both predictive distributions and stand-alone point forecasts. In a nutshell, a prediction - distributional or single-valued - is conditionally T-calibrated if it can be taken at face value in terms of the functional T. Whenever T is defined via an identification function - as in the cases of threshold (non) exceedance probabilities, quantiles, expectiles, and moments - auto-calibration implies T-calibration. We introduce population versions of T-reliability diagrams and revisit a score decomposition into measures of miscalibration (MCB), discrimination (DSC), and uncertainty (UNC). In empirical settings, stable and efficient estimators of T-reliability diagrams and score components arise via nonparametric isotonic regression and the pool-adjacent-violators algorithm. For in-sample model diagnostics, we propose a universal coefficient of determination, $$\text{R}^\ast = \frac{\text{DSC}-\text{MCB}}{\text{UNC}},$$ that nests and reinterprets the classical $\text{R}^2$ in least squares (mean) regression and its natural analogue $\text{R}^1$ in quantile regression, yet applies to T-regression in general, with MCB $\geq 0$, DSC $\geq 0$, and $\text{R}^\ast \in [0,1]$ under modest conditions.

Auto-Calibration Tests for Discrete Finite Regression Functions

Testing For Auto-Calibration With Lorenz And Concentration Curves

Consistent Specification Testing Via Nonparametric Series Regression

CHARACTERISTIC FUNCTION BASED TESTING FOR CONDITIONAL INDEPENDENCE: A NONPARAMETRIC REGRESSION APPROACH

Detecting Misspecifications in Autoregressive Conditional Duration Models

Testing Calibration in Nearly-Linear Time

Diagnostic Checking for the Adequacy of Nonlinear Time Series Models

Characteristic Function-Based Testing for Multifactor Continuous-Time Markov Models Via Nonparametric Regression

Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination

Approach to Testing Nonlinear Time Series Models

Calibration by Distribution Matching: Trainable Kernel Calibration Metrics

Size and power properties of some tests in the Birnbaum-Saunders regression model

Asymptotically Distribution-Free Goodness-of-Fit Testing: A Unifying View

A Joint Test for Parametric Specification and Independence in Nonlinear Regression Models

Specification Test based on Convolution-type Distribution Function Estimates for Non-linear Auto-regressive Processes

Empirical likelihood ratio test for autocorrelation in least squares regression

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

Conditional Independence Testing for Discrete Distributions: Beyond $χ^2$- and $G$-tests

A hypothesis test for the long-term calibration in rating systems with overlapping time windows

Goodness-of-fit Testing in Linear Regression Models

Properties of Test Statistics for Nonparametric Cointegrating Regression Functions Based on Subsamples