Abstract:Infinitely wide or deep neural networks (NNs) with independent and identically distributed (i.i.d.) parameters have been shown to be equivalent to Gaussian processes. Because of the favorable properties of Gaussian processes, this equivalence is commonly employed to analyze neural networks and has led to various breakthroughs over the years. However, neural networks and Gaussian processes are equivalent only in the limit; in the finite case there are currently no methods available to approximate a trained neural network with a Gaussian model with bounds on the approximation error. In this work, we present an algorithmic framework to approximate a neural network of finite width and depth, and with not necessarily i.i.d. parameters, with a mixture of Gaussian processes with error bounds on the approximation error. In particular, we consider the Wasserstein distance to quantify the closeness between probabilistic models and, by relying on tools from optimal transport and Gaussian processes, we iteratively approximate the output distribution of each layer of the neural network as a mixture of Gaussian processes. Crucially, for any NN and $\epsilon >0$ our approach is able to return a mixture of Gaussian processes that is $\epsilon$-close to the NN at a finite set of input points. Furthermore, we rely on the differentiability of the resulting error bound to show how our approach can be employed to tune the parameters of a NN to mimic the functional behavior of a given Gaussian process, e.g., for prior selection in the context of Bayesian inference. We empirically investigate the effectiveness of our results on both regression and classification problems with various neural network architectures. Our experiments highlight how our results can represent an important step towards understanding neural network predictions and formally quantifying their uncertainty.

Function Approximation with Spiked Random Networks

Function Approximation by Random Neural Networks with a Bounded Number of Layers

Approximation by random networks with bounded number of layers

Random Vector Functional Link Networks for Function Approximation on Manifolds

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

A More Accurate Approximation of Activation Function with Few Spikes Neurons

Random Projection Neural Networks of Best Approximation: Convergence theory and practical applications

Approximation Capability of a Novel Neural Network Model for Dynamic Systems

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Approximating nonlinear functions with latent boundaries in low-rank excitatory-inhibitory spiking networks

On the approximation by single hidden layer feedforward neural networks with fixed weights

Universal Approximation Capability of Ebf Neural Networks with Arbitrary Activation Functions

Fuzzy RBF Neural Network Based Function Approximation

Universal approximation property of Banach space-valued random feature models including random neural networks

Universal Approximation Theorem for Neural Networks

Function Gradient Approximation with Random Shallow ReLU Networks with Control Applications

Theoretically Provable Spiking Neural Networks

Efficient Estimation of Directed Connectivity in Nonlinear and Nonstationary Spiking Neuron Networks.

Back-engineering of spiking neural networks parameters

Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection