Abstract:As neural networks become more popular, the need for accompanying uncertainty estimates increases. There are currently two main approaches to test the quality of these estimates. Most methods output a density. They can be compared by evaluating their loglikelihood on a test set. Other methods output a prediction interval directly. These methods are often tested by examining the fraction of test points that fall inside the corresponding prediction intervals. Intuitively, both approaches seem logical. However, we demonstrate through both theoretical arguments and simulations that both ways of evaluating the quality of uncertainty estimates have serious flaws. Firstly, both approaches cannot disentangle the separate components that jointly create the predictive uncertainty, making it difficult to evaluate the quality of the estimates of these components. Specifically, the quality of a confidence interval cannot reliably be tested by estimating the performance of a prediction interval. Secondly, the loglikelihood does not allow a comparison between methods that output a prediction interval directly and methods that output a density. A better loglikelihood also does not necessarily guarantee better prediction intervals, which is what the methods are often used for in practice. Moreover, the current approach to test prediction intervals directly has additional flaws. We show why testing a prediction or confidence interval on a single test set is fundamentally flawed. At best, marginal coverage is measured, implicitly averaging out overconfident and underconfident predictions. A much more desirable property is pointwise coverage, requiring the correct coverage for each prediction. We demonstrate through practical examples that these effects can result in favouring a method, based on the predictive uncertainty, that has undesirable behaviour of the confidence or prediction intervals. Finally, we propose a simulation-based testing approach that addresses these problems while still allowing easy comparison between different methods. This approach can be used for the development of new uncertainty quantification methods.

Valid predictions of random quantities in linear mixed models

Linear mixed effects models for non‐Gaussian continuous repeated measurement data

Approximate Inferences for Nonlinear Mixed Effects Models with Scale Mixtures of Skew-Normal Distributions

Mixed Linear Model Approaches for Analyzing Genetic Models of Complex Quantitative Traits

Uncertainty Quantification in Synthetic Controls with Staggered Treatment Adoption

Estimating individual treatment effect: generalization bounds and algorithms

Marginalization in nonlinear mixed-effects models with an application to dose-response analysis

Uncertainty Computation at Finite Distance in Nonlinear Mixed Effects Models—a New Method Based on Metropolis-Hastings Algorithm

Estimation for stochastic differential equation mixed models using approximation methods

Uncertainty Quantification in Heterogeneous Treatment Effect Estimation with Gaussian-Process-Based Partially Linear Model

Building Conformal Prediction Intervals with Approximate Message Passing

Uncertainty quantification for intervals

Inference for High-Dimensional Linear Mixed-Effects Models: A Quasi-Likelihood Approach

How to evaluate uncertainty estimates in machine learning for regression?

Large-sample theory for inferential models: a possibilistic Bernstein--von Mises theorem

Valid prediction intervals for regression problems

Optimal Design of Validation Experiments for the Prediction of Quantities of Interest

Uncertainty Quantification of Surrogate Models using Conformal Prediction

A new analysis of optimal estimation and prediction under linear mixed models

Improved confidence intervals for nonlinear mixed-effects and nonparametric regression models

Likelihood Function through the Delta Approximation in Mixed SDE Models