Abstract:Neal (1996) proved that infinitely wide shallow Bayesian neural networks (BNN) converge to Gaussian processes (GP), when the network weights have bounded prior variance. Cho & Saul (2009) provided a useful recursive formula for deep kernel processes for relating the covariance kernel of each layer to the layer immediately below. Moreover, they worked out the form of the layer-wise covariance kernel in an explicit manner for several common activation functions. Recent works, including Aitchison et al. (2021), have highlighted that the covariance kernels obtained in this manner are deterministic and hence, precludes any possibility of representation learning, which amounts to learning a non-degenerate posterior of a random kernel given the data. To address this, they propose adding artificial noise to the kernel to retain stochasticity, and develop deep kernel inverse Wishart processes. Nonetheless, this artificial noise injection could be critiqued in that it would not naturally emerge in a classic BNN architecture under an infinite-width limit. To address this, we show that a Bayesian deep neural network, where each layer width approaches infinity, and all network weights are elliptically distributed with infinite variance, converges to a process with $\alpha$-stable marginals in each layer that has a conditionally Gaussian representation. These conditional random covariance kernels could be recursively linked in the manner of Cho & Saul (2009), even though marginally the process exhibits stable behavior, and hence covariances are not even necessarily defined. We also provide useful generalizations of the recent results of Loría & Bhadra (2024) on shallow networks to multi-layer networks, and remedy the computational burden of their approach. The computational and statistical benefits over competing approaches stand out in simulations and in demonstrations on benchmark data sets.

Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case

An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network

Invariance of Weight Distributions in Rectified MLPs

A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability

Free Probability for predicting the performance of feed-forward fully connected neural networks

Deep Kernel Posterior Learning under Infinite Variance Prior Weights

Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity

The Proof of Kolmogorov-Arnold May Illuminate Neural Network Learning

Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignment

Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications

Hessian Eigenvectors and Principal Component Analysis of Neural Network Weight Matrices

Algebraic geometrical methods for hierarchical learning machines

On the Compatibility between Neural Networks and Partial Differential Equations for Physics-informed Learning

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks

Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility

Proportional infinite-width infinite-depth limit for deep linear neural networks

Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff

Estimating the Jacobian matrix of an unknown multivariate function from sample values by means of a neural network

On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case

Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks

Representing Neural Network Layers as Linear Operations via Koopman Operator Theory