Abstract:Infinitely wide or deep neural networks (NNs) with independent and identically distributed (i.i.d.) parameters have been shown to be equivalent to Gaussian processes. Because of the favorable properties of Gaussian processes, this equivalence is commonly employed to analyze neural networks and has led to various breakthroughs over the years. However, neural networks and Gaussian processes are equivalent only in the limit; in the finite case there are currently no methods available to approximate a trained neural network with a Gaussian model with bounds on the approximation error. In this work, we present an algorithmic framework to approximate a neural network of finite width and depth, and with not necessarily i.i.d. parameters, with a mixture of Gaussian processes with error bounds on the approximation error. In particular, we consider the Wasserstein distance to quantify the closeness between probabilistic models and, by relying on tools from optimal transport and Gaussian processes, we iteratively approximate the output distribution of each layer of the neural network as a mixture of Gaussian processes. Crucially, for any NN and $\epsilon >0$ our approach is able to return a mixture of Gaussian processes that is $\epsilon$-close to the NN at a finite set of input points. Furthermore, we rely on the differentiability of the resulting error bound to show how our approach can be employed to tune the parameters of a NN to mimic the functional behavior of a given Gaussian process, e.g., for prior selection in the context of Bayesian inference. We empirically investigate the effectiveness of our results on both regression and classification problems with various neural network architectures. Our experiments highlight how our results can represent an important step towards understanding neural network predictions and formally quantifying their uncertainty.

Fourier Neural Network Approximation of Transition Densities in Finance

Fourier Neural Networks for Function Approximation

Approximating Numerical Fluxes Using Fourier Neural Operators for Hyperbolic Conservation Laws

Neural Fourier Modelling: A Highly Compact Approach to Time-Series Analysis

Fourier neural networks as function approximators and differential equation solvers

Deep learning of transition probability densities for stochastic asset models with applications in option pricing

Deep Learning without Global Optimization by Random Fourier Neural Networks

Fourier warm start for physics-informed neural networks

Simo Fourier Neural Networks Research

Physics-embedded Fourier Neural Network for Partial Differential Equations

Neural network stochastic differential equation models with applications to financial data forecasting

Fourier Head: Helping Large Language Models Learn Complex Probability Distributions

Discretization Error of Fourier Neural Operators

Fourier-Net: Fast Image Registration with Band-limited Deformation

Robust Fourier Neural Networks

Neural network representation of the probability density function of diffusion processes

Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection

Understanding Training and Generalization in Deep Learning by Fourier Analysis.

Physics-informed Neural Networks with Fourier Features for Seismic Wavefield Simulation in Time-Domain Nonsmooth Complex Media

Fourier Basis Density Model

Enforcing asymptotic behavior with DNNs for approximation and regression in finance