Abstract:Neal (1996) proved that infinitely wide shallow Bayesian neural networks (BNN) converge to Gaussian processes (GP), when the network weights have bounded prior variance. Cho & Saul (2009) provided a useful recursive formula for deep kernel processes for relating the covariance kernel of each layer to the layer immediately below. Moreover, they worked out the form of the layer-wise covariance kernel in an explicit manner for several common activation functions. Recent works, including Aitchison et al. (2021), have highlighted that the covariance kernels obtained in this manner are deterministic and hence, precludes any possibility of representation learning, which amounts to learning a non-degenerate posterior of a random kernel given the data. To address this, they propose adding artificial noise to the kernel to retain stochasticity, and develop deep kernel inverse Wishart processes. Nonetheless, this artificial noise injection could be critiqued in that it would not naturally emerge in a classic BNN architecture under an infinite-width limit. To address this, we show that a Bayesian deep neural network, where each layer width approaches infinity, and all network weights are elliptically distributed with infinite variance, converges to a process with $\alpha$-stable marginals in each layer that has a conditionally Gaussian representation. These conditional random covariance kernels could be recursively linked in the manner of Cho & Saul (2009), even though marginally the process exhibits stable behavior, and hence covariances are not even necessarily defined. We also provide useful generalizations of the recent results of Loría & Bhadra (2024) on shallow networks to multi-layer networks, and remedy the computational burden of their approach. The computational and statistical benefits over competing approaches stand out in simulations and in demonstrations on benchmark data sets.

Deep learning for ψ -weakly dependent processes

Excess risk bound for deep learning under weak dependence

Penalized deep neural networks estimator with general loss functions under weak dependence

Robust deep learning from weakly dependent data

Sparse-penalized deep neural networks estimator under weak dependence

Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality

Trade-off Between Dependence and Complexity for Nonparametric Learning -- an Empirical Process Approach

Sharp Rates in Dependent Learning Theory: Avoiding Sample Size Deflation for the Square Loss

Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility

Statistical Properties of Deep Neural Networks with Dependent Data

Theoretical analysis of deep neural networks for temporally dependent observations

A generalization bound of deep neural networks for dependent data

Predicting path-dependent processes by deep learning

Deep Kernel Posterior Learning under Infinite Variance Prior Weights

Learning from weakly dependent data under Dobrushin's condition

Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?

Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds

Over-parameterized Deep Nonparametric Regression for Dependent Data with Its Applications to Reinforcement Learning.

A deep learning approach to the probabilistic numerical solution of path-dependent partial differential equations

Learning with little mixing