Abstract:Large neural networks trained on large datasets have become the dominant paradigm in machine learning. These systems rely on maximum likelihood point estimates of their parameters, precluding them from expressing model uncertainty. This may result in overconfident predictions and it prevents the use of deep learning models for sequential decision making. This thesis develops scalable methods to equip neural networks with model uncertainty. In particular, we leverage the linearised Laplace approximation to equip pre-trained neural networks with the uncertainty estimates provided by their tangent linear models. This turns the problem of Bayesian inference in neural networks into one of Bayesian inference in conjugate Gaussian-linear models. Alas, the cost of this remains cubic in either the number of network parameters or in the number of observations times output dimensions. By assumption, neither are tractable. We address this intractability by using stochastic gradient descent (SGD) -- the workhorse algorithm of deep learning -- to perform posterior sampling in linear models and their convex duals: Gaussian processes. With this, we turn back to linearised neural networks, finding the linearised Laplace approximation to present a number of incompatibilities with modern deep learning practices -- namely, stochastic optimisation, early stopping and normalisation layers -- when used for hyperparameter learning. We resolve these and construct a sample-based EM algorithm for scalable hyperparameter learning with linearised neural networks. We apply the above methods to perform linearised neural network inference with ResNet-50 (25M parameters) trained on Imagenet (1.2M observations and 1000 output dimensions). Additionally, we apply our methods to estimate uncertainty for 3d tomographic reconstructions obtained with the deep image prior network.

A Comprehensive Study of Calibration and Uncertainty Quantification for Bayesian Convolutional Neural Networks - An Application to Seismic Data

Calibration and Uncertainty Quantification of Bayesian Convolutional Neural Networks for Geophysical Applications

Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification

Combining unsupervised deep learning and Monte Carlo dropout for seismic data reconstruction and its uncertainty quantification

Accurate Uncertainties for Deep Learning Using Calibrated Regression

Uncertainty Quantification for Deep Learning

Towards calibrated and scalable uncertainty representations for neural networks

Neural State-Space Models: Empirical Evaluation of Uncertainty Quantification

Calibration-Aware Bayesian Learning

Calibrating Ensembles for Scalable Uncertainty Quantification in Deep Learning-based Medical Segmentation

Uncertainty Quantification in Seismic Inversion Through Integrated Importance Sampling and Ensemble Methods

Uncertainty Quantification in Inverse Scattering Problems With Bayesian Convolutional Neural Networks

Leveraging the Bhattacharyya coefficient for uncertainty quantification in deep neural networks

Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization

On the Calibration of Epistemic Uncertainty: Principles, Paradoxes and Conflictual Loss

Beyond Calibration: Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence

Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures

Uncertainty estimation via ensembles of deep learning models and dropout layers for seismic traces

Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms

Improving model calibration with accuracy versus uncertainty optimization

Scalable Bayesian Inference in the Era of Deep Learning: From Gaussian Processes to Deep Neural Networks