Abstract:On top of machine learning (ML) models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability improvement of ML models empowered by UQ has the potential to significantly facilitate the broad adoption of ML solutions in high-stakes decision settings, such as healthcare, manufacturing, and aviation, to name a few. In this tutorial, we aim to provide a holistic lens on emerging UQ methods for ML models with a particular focus on neural networks and the applications of these UQ methods in tackling engineering design as well as prognostics and health management problems. Towards this goal, we start with a comprehensive classification of uncertainty types, sources, and causes pertaining to UQ of ML models. Next, we provide a tutorial-style description of several state-of-the-art UQ methods: Gaussian process regression, Bayesian neural network, neural network ensemble, and deterministic UQ methods focusing on spectral-normalized neural Gaussian process. Established upon the mathematical formulations, we subsequently examine the soundness of these UQ methods quantitatively and qualitatively (by a toy regression example) to examine their strengths and shortcomings from different dimensions. Then, we review quantitative metrics commonly used to assess the quality of predictive uncertainty in classification and regression problems. Afterward, we discuss the increasingly important role of UQ of ML models in solving challenging problems in engineering design and health prognostics. Two case studies with source codes available on GitHub are used to demonstrate these UQ methods and compare their performance in the life prediction of lithium-ion batteries at the early stage (case study 1) and the remaining useful life prediction of turbofan engines (case study 2).

UNIQUE: A Framework for Uncertainty Quantification Benchmarking

Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons

A hybrid framework for improving uncertainty quantification in deep learning-based QSAR regression modeling

Uncertainty-Based Rejection in Machine Learning: Implications for Model Development and Interpretability

NeuralUQ: A Comprehensive Library for Uncertainty Quantification in Neural Differential Equations and Operators

Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AI

Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial

Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity concepts

Evaluating and Boosting Uncertainty Quantification in Classification

Uncertainty Quantification Driven Machine Learning for Improving Model Accuracy in Imbalanced Regression Tasks

Assessment of Uncertainty Quantification in Universal Differential Equations

A Meta-heuristic Approach to Estimate and Explain Classifier Uncertainty

Uncertainty Quantification Using Neural Networks for Molecular Property Prediction

Democratizing Uncertainty Quantification

Statistical methods for resolving poor uncertainty quantification in machine learning interatomic potentials

Legitimate ground-truth-free metrics for deep uncertainty classification scoring

Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry

MUBen: Benchmarking the Uncertainty of Molecular Representation Models

A Survey on Uncertainty Quantification Methods for Deep Learning

Uncertainty Quantification in Machine Learning for Biosignal Applications -- A Review