Abstract:Reliable spatial uncertainty evaluation of object detection models is of special interest and has been subject of recent work. In this work, we review the existing definitions for uncertainty calibration of probabilistic regression tasks. We inspect the calibration properties of common detection networks and extend state-of-the-art recalibration methods. Our methods use a Gaussian process (GP) recalibration scheme that yields parametric distributions as output (e.g. Gaussian or Cauchy). The usage of GP recalibration allows for a local (conditional) uncertainty calibration by capturing dependencies between neighboring samples. The use of parametric distributions such as as Gaussian allows for a simplified adaption of calibration in subsequent processes, e.g., for Kalman filtering in the scope of object tracking. In addition, we use the GP recalibration scheme to perform covariance estimation which allows for post-hoc introduction of local correlations between the output quantities, e.g., position, width, or height in object detection. To measure the joint calibration of multivariate and possibly correlated data, we introduce the quantile calibration error which is based on the Mahalanobis distance between the predicted distribution and the ground truth to determine whether the ground truth is within a predicted quantile. Our experiments show that common detection models overestimate the spatial uncertainty in comparison to the observed error. We show that the simple Isotonic Regression recalibration method is sufficient to achieve a good uncertainty quantification in terms of calibrated quantiles. In contrast, if normal distributions are required for subsequent processes, our GP-Normal recalibration method yields the best results. Finally, we show that our covariance estimation method is able to achieve best calibration results for joint multivariate calibration.

Learning Uncertainties the Frequentist Way: Calibration and Correlation in High Energy Physics

Seeing Double: Calibrating Two Jets at Once

Calibrating two jets at once

Accurate Uncertainties for Deep Learning Using Calibrated Regression

Matching Bayesian and frequentist coverage probabilities when using an approximate data covariance matrix

Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection

A Comprehensive Study of Calibration and Uncertainty Quantification for Bayesian Convolutional Neural Networks - An Application to Seismic Data

Robust and scalable uncertainty estimation with conformal prediction for machine-learned interatomic potentials

Calibrating Bayesian Generative Machine Learning for Bayesiamplification

On the Calibration of Epistemic Uncertainty: Principles, Paradoxes and Conflictual Loss

Uncertainty Aware Deep Learning for Particle Accelerators

Calibration and Uncertainty Quantification of Bayesian Convolutional Neural Networks for Geophysical Applications

Evaluating and Calibrating Uncertainty Prediction in Regression Tasks

Beyond Calibration: Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence

Calibrated Uncertainty Quantification for Operator Learning via Conformal Prediction

Using Machine Learning to Improve PDF Uncertainties

Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures

New metric improving Bayesian calibration of a multistage approach studying hadron and inclusive jet suppression

Jet energy calibration with deep learning as a Kubeflow pipeline

Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression tasks

Uncertainty Quantification for Nuclear Density Functional Theory and Information Content of New Measurements