Abstract:Infinitely wide or deep neural networks (NNs) with independent and identically distributed (i.i.d.) parameters have been shown to be equivalent to Gaussian processes. Because of the favorable properties of Gaussian processes, this equivalence is commonly employed to analyze neural networks and has led to various breakthroughs over the years. However, neural networks and Gaussian processes are equivalent only in the limit; in the finite case there are currently no methods available to approximate a trained neural network with a Gaussian model with bounds on the approximation error. In this work, we present an algorithmic framework to approximate a neural network of finite width and depth, and with not necessarily i.i.d. parameters, with a mixture of Gaussian processes with error bounds on the approximation error. In particular, we consider the Wasserstein distance to quantify the closeness between probabilistic models and, by relying on tools from optimal transport and Gaussian processes, we iteratively approximate the output distribution of each layer of the neural network as a mixture of Gaussian processes. Crucially, for any NN and $\epsilon >0$ our approach is able to return a mixture of Gaussian processes that is $\epsilon$-close to the NN at a finite set of input points. Furthermore, we rely on the differentiability of the resulting error bound to show how our approach can be employed to tune the parameters of a NN to mimic the functional behavior of a given Gaussian process, e.g., for prior selection in the context of Bayesian inference. We empirically investigate the effectiveness of our results on both regression and classification problems with various neural network architectures. Our experiments highlight how our results can represent an important step towards understanding neural network predictions and formally quantifying their uncertainty.

An Overview of Uncertainty Quantification Methods for Infinite Neural Networks

Stochastic Neural Networks with Infinite Width are Deterministic

Uncertainty Quantification of Graph Convolution Neural Network Models of Evolving Processes

Posterior Inference on Shallow Infinitely Wide Bayesian Neural Networks under Weights with Unbounded Variance

Uncertainty Quantification for Deep Learning

Uncertainty Quantification for Sparse Deep Learning

Neural State-Space Models: Empirical Evaluation of Uncertainty Quantification

Generalized Uncertainty of Deep Neural Networks: Taxonomy and Applications

Uncertainty Quantification in Deep Neural Networks through Statistical Inference on Latent Space

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training

A Kernel Framework to Quantify a Model's Local Predictive Uncertainty under Data Distributional Shifts

A Unified Theory of Quantum Neural Network Loss Landscapes

Approaching Neural Network Uncertainty Realism

Evaluating Uncertainty Quantification approaches for Neural PDEs in scientific applications

A General Framework for quantifying Aleatoric and Epistemic uncertainty in Graph Neural Networks

Uncertainty Quantification using Generative Approach

Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization

Uncertainty quantification for noisy inputs-outputs in physics-informed neural networks and neural operators

Deep Neural Networks as Gaussian Processes

Parameterizing uncertainty by deep invertible networks, an application to reservoir characterization