Single-shot Bayesian approximation for neural networks

Kai Brach,Beate Sick,Oliver Dürr

2023-08-24

Abstract:Deep neural networks (NNs) are known for their high-prediction performances. However, NNs are prone to yield unreliable predictions when encountering completely new situations without indicating their uncertainty. Bayesian variants of NNs (BNNs), such as Monte Carlo (MC) dropout BNNs, do provide uncertainty measures and simultaneously increase the prediction performance. The only disadvantage of BNNs is their higher computation time during test time because they rely on a sampling approach. Here we present a single-shot MC dropout approximation that preserves the advantages of BNNs while being as fast as NNs. Our approach is based on moment propagation (MP) and allows to analytically approximate the expected value and the variance of the MC dropout signal for commonly used layers in NNs, i.e. convolution, max pooling, dense, softmax, and dropout layers. The MP approach can convert an NN into a BNN without re-training given the NN has been trained with standard dropout. We evaluate our approach on different benchmark datasets and a simulated toy example in a classification and regression setting. We demonstrate that our single-shot MC dropout approximation resembles the point estimate and the uncertainty estimate of the predictive distribution that is achieved with an MC approach, while being fast enough for real-time deployments of BNNs. We show that using part of the saved time to combine our MP approach with deep ensemble techniques does further improve the uncertainty measures.

Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the problem of achieving efficient uncertainty estimation in neural networks (NN). Specifically, traditional neural networks, while having high predictive performance, cannot reliably indicate their uncertainty when encountering entirely new situations. Bayesian neural networks (BNN) can provide uncertainty measures and improve predictive performance, but they require sampling methods during the testing phase, leading to longer computation times and limiting their use in real-time applications. The paper proposes a single-shot MC dropout approximation method, which can estimate the expected value and variance of the predictive distribution with computation time comparable to traditional neural networks, without sacrificing the advantages of Bayesian neural networks. This method is based on Moment Propagation (MP) and can analyze the expected value and variance of signal distributions in common neural network layers such as convolutional, max-pooling, fully connected, Softmax, and Dropout layers, thereby converting a traditional neural network into a Bayesian neural network without retraining. In this way, the method can provide reliable uncertainty estimation while maintaining fast response, making it suitable for real-time applications such as autonomous driving.