Abstract:Quantifying uncertainty of predictions has been identified as one way to develop more trustworthy artificial intelligence (AI) models beyond conventional reporting of performance metrics. When considering their role in a clinical decision support setting, AI classification models should ideally avoid confident wrong predictions and maximise the confidence of correct predictions. Models that do this are said to be well-calibrated with regard to confidence. However, relatively little attention has been paid to how to improve calibration when training these models, i.e., to make the training strategy uncertainty-aware. In this work we evaluate three novel uncertainty-aware training strategies comparing against two state-of-the-art approaches. We analyse performance on two different clinical applications: cardiac resynchronisation therapy (CRT) response prediction and coronary artery disease (CAD) diagnosis from cardiac magnetic resonance (CMR) images. The best-performing model in terms of both classification accuracy and the most common calibration measure, expected calibration error (ECE) was the Confidence Weight method, a novel approach that weights the loss of samples to explicitly penalise confident incorrect predictions. The method reduced the ECE by 17% for CRT response prediction and by 22% for CAD diagnosis when compared to a baseline classifier in which no uncertainty-aware strategy was included. In both applications, as well as reducing the ECE there was a slight increase in accuracy from 69% to 70% and 70% to 72% for CRT response prediction and CAD diagnosis respectively. However, our analysis showed a lack of consistency in terms of optimal models when using different calibration measures. This indicates the need for careful consideration of performance metrics when training and selecting models for complex high-risk applications in healthcare.

Test Time Augmentation Meets Post-hoc Calibration: Uncertainty Quantification under Real-World Conditions

Post-hoc Uncertainty Calibration for Domain Drift Scenarios

Adaptive temperature scaling for Robust calibration of deep neural networks

Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration

Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization

Accurate Uncertainties for Deep Learning Using Calibrated Regression

Calibrating Deep Neural Network using Euclidean Distance

A Comprehensive Study of Calibration and Uncertainty Quantification for Bayesian Convolutional Neural Networks - An Application to Seismic Data

Uncertainty Calibration with Energy Based Instance-wise Scaling in the Wild Dataset

Improving model calibration with accuracy versus uncertainty optimization

Test-time Augmentation with Uncertainty Estimation for Deep Learning-Based Medical Image Segmentation

Uncertainty Aware Training to Improve Deep Learning Model Calibration for Classification of Cardiac MR Images

Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors

Revisiting the Calibration of Modern Neural Networks

Calibrating Ensembles for Scalable Uncertainty Quantification in Deep Learning-based Medical Segmentation

Beyond Calibration: Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence

Towards calibrated and scalable uncertainty representations for neural networks

Calibration and Uncertainty Quantification of Bayesian Convolutional Neural Networks for Geophysical Applications

Verified Uncertainty Calibration

End-to-End Conformal Calibration for Optimization Under Uncertainty

Addressing Deep Learning Model Calibration Using Evidential Neural Networks And Uncertainty-Aware Training