Abstract:Neal (1996) proved that infinitely wide shallow Bayesian neural networks (BNN) converge to Gaussian processes (GP), when the network weights have bounded prior variance. Cho & Saul (2009) provided a useful recursive formula for deep kernel processes for relating the covariance kernel of each layer to the layer immediately below. Moreover, they worked out the form of the layer-wise covariance kernel in an explicit manner for several common activation functions. Recent works, including Aitchison et al. (2021), have highlighted that the covariance kernels obtained in this manner are deterministic and hence, precludes any possibility of representation learning, which amounts to learning a non-degenerate posterior of a random kernel given the data. To address this, they propose adding artificial noise to the kernel to retain stochasticity, and develop deep kernel inverse Wishart processes. Nonetheless, this artificial noise injection could be critiqued in that it would not naturally emerge in a classic BNN architecture under an infinite-width limit. To address this, we show that a Bayesian deep neural network, where each layer width approaches infinity, and all network weights are elliptically distributed with infinite variance, converges to a process with $\alpha$-stable marginals in each layer that has a conditionally Gaussian representation. These conditional random covariance kernels could be recursively linked in the manner of Cho & Saul (2009), even though marginally the process exhibits stable behavior, and hence covariances are not even necessarily defined. We also provide useful generalizations of the recent results of Loría & Bhadra (2024) on shallow networks to multi-layer networks, and remedy the computational burden of their approach. The computational and statistical benefits over competing approaches stand out in simulations and in demonstrations on benchmark data sets.

Infinite‐width limit of deep linear neural networks

Stochastic Neural Networks with Infinite Width are Deterministic

Proportional infinite-width infinite-depth limit for deep linear neural networks

Dynamically Stable Infinite-Width Limits of Neural Classifiers

On the infinite width limit of neural networks with a standard parameterization

Towards a General Theory of Infinite-Width Limits of Neural Classifiers

An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network

Doubly infinite residual neural networks: a diffusion process approach

Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functions

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training

Deep Kernel Posterior Learning under Infinite Variance Prior Weights

Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility

Dynamics of finite width Kernel and prediction fluctuations in mean field neural networks *

Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks

Deep Limits of Residual Neural Networks

Infinite Width Limits of Self Supervised Neural Networks

Feature Learning in Infinite-Width Neural Networks

Width and Depth Limits Commute in Residual Networks

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization

Dynamics of Deep Neural Networks and Neural Tangent Hierarchy

Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks