Abstract:Infinitely wide or deep neural networks (NNs) with independent and identically distributed (i.i.d.) parameters have been shown to be equivalent to Gaussian processes. Because of the favorable properties of Gaussian processes, this equivalence is commonly employed to analyze neural networks and has led to various breakthroughs over the years. However, neural networks and Gaussian processes are equivalent only in the limit; in the finite case there are currently no methods available to approximate a trained neural network with a Gaussian model with bounds on the approximation error. In this work, we present an algorithmic framework to approximate a neural network of finite width and depth, and with not necessarily i.i.d. parameters, with a mixture of Gaussian processes with error bounds on the approximation error. In particular, we consider the Wasserstein distance to quantify the closeness between probabilistic models and, by relying on tools from optimal transport and Gaussian processes, we iteratively approximate the output distribution of each layer of the neural network as a mixture of Gaussian processes. Crucially, for any NN and $\epsilon >0$ our approach is able to return a mixture of Gaussian processes that is $\epsilon$-close to the NN at a finite set of input points. Furthermore, we rely on the differentiability of the resulting error bound to show how our approach can be employed to tune the parameters of a NN to mimic the functional behavior of a given Gaussian process, e.g., for prior selection in the context of Bayesian inference. We empirically investigate the effectiveness of our results on both regression and classification problems with various neural network architectures. Our experiments highlight how our results can represent an important step towards understanding neural network predictions and formally quantifying their uncertainty.

Convergence of neural networks to Gaussian mixture distribution

Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection

Proportional infinite-width infinite-depth limit for deep linear neural networks

Gaussian Universality in Neural Network Dynamics with Generalized Structured Input Distributions

Deep quantum neural networks form Gaussian processes

Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility

Quantitative convergence of trained quantum neural networks to a Gaussian process

Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processes

Deep Neural Networks as Gaussian Processes

On the Convergence Rate of Gaussianization with Random Rotations

Statistical Convergence of the EM Algorithm on Gaussian Mixture Models

Toward Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixture Models

Learning nonlinear manifolds based on mixtures of localized linear manifolds under a self-organizing framework

Neural Network Gaussian Processes by Increasing Depth

A Convergence Rate for Manifold Neural Networks

Convergence of Message Passing Graph Neural Networks with Generic Aggregation On Large Random Graphs

Random ReLU Neural Networks as Non-Gaussian Processes

Wide Neural Networks with Bottlenecks are Deep Gaussian Processes

Sharp Convergence of Nonlinear Functionals of a Class of Gaussian Random Fields

Normal approximation of Random Gaussian Neural Networks

Convergence rates of non-stationary and deep Gaussian process regression