Abstract:As neural networks become more popular, the need for accompanying uncertainty estimates increases. There are currently two main approaches to test the quality of these estimates. Most methods output a density. They can be compared by evaluating their loglikelihood on a test set. Other methods output a prediction interval directly. These methods are often tested by examining the fraction of test points that fall inside the corresponding prediction intervals. Intuitively, both approaches seem logical. However, we demonstrate through both theoretical arguments and simulations that both ways of evaluating the quality of uncertainty estimates have serious flaws. Firstly, both approaches cannot disentangle the separate components that jointly create the predictive uncertainty, making it difficult to evaluate the quality of the estimates of these components. Specifically, the quality of a confidence interval cannot reliably be tested by estimating the performance of a prediction interval. Secondly, the loglikelihood does not allow a comparison between methods that output a prediction interval directly and methods that output a density. A better loglikelihood also does not necessarily guarantee better prediction intervals, which is what the methods are often used for in practice. Moreover, the current approach to test prediction intervals directly has additional flaws. We show why testing a prediction or confidence interval on a single test set is fundamentally flawed. At best, marginal coverage is measured, implicitly averaging out overconfident and underconfident predictions. A much more desirable property is pointwise coverage, requiring the correct coverage for each prediction. We demonstrate through practical examples that these effects can result in favouring a method, based on the predictive uncertainty, that has undesirable behaviour of the confidence or prediction intervals. Finally, we propose a simulation-based testing approach that addresses these problems while still allowing easy comparison between different methods. This approach can be used for the development of new uncertainty quantification methods.

Using random set theory to propagate epistemic uncertainty through a mechanical system

Unified Reliability Measure Method Considering Uncertainties of Input Variables and Their Distribution Parameters

Random Fuzzy Extension of the Universal Generating Function Approach for the Reliability Assessment of Multi-State Systems under Aleatory and Epistemic Uncertainties

Fighting Uncertainty with Uncertainty: A Baby Step

Parameter estimation for mechanical systems via an explicit representation of uncertainty

A systematic review of uncertainty theory with the use of scientometrical method

Approximate Propagation of both Epistemic and Aleatory Uncertainty through Dynamic Systems

Sensitivity Analysis of Random and Interval Uncertain Variables Based on Polynomial Chaos Expansion Method

Polynomial Chaos Surrogate Construction for Random Fields with Parametric Uncertainty

A probabilistic simulation method for sensitivity analysis of input epistemic uncertainties on failure probability

Random time step probabilistic methods for uncertainty quantification in chaotic and geometric numerical integration

A collaborative quasi-Monte Carlo uncertainty propagation analysis method for multiple types of epistemic uncertainty quantified by probability boxes

Scientific Programs Imply Uncertainty. Results Expected and Unexpected

Assessment of Uncertainty Quantification in Universal Differential Equations

Probabilistic feasibility guarantees for solution sets to uncertain variational inequalities

How to evaluate uncertainty estimates in machine learning for regression?

Taming uncertainty with MDE: an historical perspective

Uncertainty measures: The big picture

Quantifying Aleatoric and Epistemic Dynamics Uncertainty via Local Conformal Calibration

Hybrid Bayesian Solution to Nasa Langley Research Center Multidisciplinary Uncertainty Quantification Challenge

A Nonparametric Bayesian Framework for Uncertainty Quantification in Stochastic Simulation