Abstract:We present $\Delta$-UQ -- a novel, general-purpose uncertainty estimator using the concept of anchoring in predictive models. Anchoring works by first transforming the input into a tuple consisting of an anchor point drawn from a prior distribution, and a combination of the input sample with the anchor using a pretext encoding scheme. This encoding is such that the original input can be perfectly recovered from the tuple -- regardless of the choice of the anchor. Therefore, any predictive model should be able to predict the target response from the tuple alone (since it implicitly represents the input). Moreover, by varying the anchors for a fixed sample, we can estimate uncertainty in the prediction even using only a single predictive model. We find this uncertainty is deeply connected to improper sampling of the input data, and inherent noise, enabling us to estimate the total uncertainty in any system. With extensive empirical studies on a variety of use-cases, we demonstrate that $\Delta$-UQ outperforms several competitive baselines. Specifically, we study model fitting, sequential model optimization, model based inversion in the regression setting and out of distribution detection, & calibration under distribution shifts for classification.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to accurately and reliably estimate uncertainty in machine - learning models**. Specifically, the paper proposes a new method named ∆-UQ, aiming to quantify the uncertainty in prediction models through the anchoring technique. This method can be applied to any machine - learning model and is suitable for classification and regression tasks. ### Core Problems of the Paper 1. **Limitations of Existing Uncertainty Estimation Methods**: - Most of the existing uncertainty estimation methods are only applicable to specific types of models or tasks (such as classification or regression), and many methods require special training schemes. - Some methods (such as Bayesian neural networks, deep ensembles, etc.) are effective but have high computational costs or are limited to specific scenarios. 2. **Requirement for Single - Model Uncertainty Estimation**: - A method that can use a single model to estimate uncertainty is needed to simplify the model architecture and reduce computational costs. - This method should be able to handle classification and regression tasks and have no strict limitations on the model type. ### ∆-UQ Solution ∆-UQ re - parameterizes the input data by introducing anchors to achieve uncertainty estimation. The specific steps are as follows: 1. **Anchoring Encoding**: - For each input sample $X$, generate a randomly selected anchor $R$. - Combine the input $X$ and the anchor $R$ into a tuple $(R,\Delta(X,R))$, where $\Delta(X,R)$ is a predefined encoding method to ensure that the original input $X$ can be recovered from the tuple. 2. **Prediction Model**: - Use the combined tuple as input to train a prediction model $f_\Delta(\theta)$ so that it can predict the target value $y$ from the tuple. - In the inference stage, the uncertainty of the prediction results can be estimated by changing different anchors $R$. 3. **Uncertainty Estimation**: - Calculate the mean and variance of the prediction results by sampling different anchors $R$ multiple times, thereby obtaining the uncertainty estimate. - These uncertainties are associated with the epistemic uncertainty due to insufficient sampling of the input data and the aleatoric uncertainty (inherent noise). ### Experimental Verification The paper verifies the effectiveness of ∆-UQ through multiple experiments, including but not limited to: - **Regression Tasks**: Uncertainty calibration for regression tasks on the OpenML dataset, and the results show that ∆-UQ performs better than other baseline methods. - **Active Learning**: In the function optimization task, use uncertainty estimation for active learning, and the results show that ∆-UQ can find the optimal solution more efficiently. - **Classification Tasks**: OOD (Out - of - Distribution) detection and calibration under distribution shift on the CIFAR - 10 and SVHN datasets, and the results show that ∆-UQ can better detect abnormal samples and maintain the calibration performance of the model. In summary, this paper aims to provide a general and efficient uncertainty estimation method to solve the limitations of existing methods in terms of applicability and computational cost.

$Δ$-UQ: Accurate Uncertainty Quantification via Anchor Marginalization

Uncertainty Quantification of the 4th kind; optimal posterior accuracy-uncertainty tradeoff with the minimum enclosing ball

Non-Asymptotic Uncertainty Quantification in High-Dimensional Learning

Optimal Uncertainty Quantification

DeepUQ: Assessing the Aleatoric Uncertainties from two Deep Learning Methods

UNIQUE: A Framework for Uncertainty Quantification Benchmarking

DiffHybrid-UQ: Uncertainty Quantification for Differentiable Hybrid Neural Modeling

Uncertainty Quantification for Deep Learning

NeuralUQ: A Comprehensive Library for Uncertainty Quantification in Neural Differential Equations and Operators

A two-step surrogate method for sequential uncertainty quantification in high-dimensional inverse problems

Uncertainty quantification for deep learning-based schemes for solving high-dimensional backward stochastic differential equations

Assessment of Uncertainty Quantification in Universal Differential Equations

A Survey on Uncertainty Quantification Methods for Deep Learning

Uncertainty Quantification in Neural Differential Equations

Taming Uncertainty in a Complex World: The Rise of Uncertainty Quantification -- A Tutorial for Beginners

Easy Uncertainty Quantification (EasyUQ): Generating Predictive Distributions from Single-Valued Model Output

Quality measures for the evaluation of machine learning architectures on the quantification of epistemic and aleatoric uncertainties in complex dynamical systems

SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models

Robust and scalable uncertainty estimation with conformal prediction for machine-learned interatomic potentials

Quantification of Uncertainty with Adversarial Models

Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms