Abstract:Modern neural networks (NN) featuring a large number of layers (depth) and units per layer (width) have achieved a remarkable performance across many domains. While there exists a vast literature on the interplay between infinitely wide NNs and Gaussian processes, a little is known about analogous interplays with respect to infinitely deep NNs. NNs with independent and identically distributed (i.i.d.) initializations exhibit undesirable forward and backward propagation properties as the number of layers increases. To overcome these drawbacks, Peluchetti and Favaro (2020) considered fully-connected residual networks (ResNets) with network's parameters initialized by means of distributions that shrink as the number of layers increases, thus establishing an interplay between infinitely deep ResNets and solutions to stochastic differential equations, i.e. diffusion processes, and showing that infinitely deep ResNets does not suffer from undesirable forward-propagation properties. In this paper, we review the results of Peluchetti and Favaro (2020), extending them to convolutional ResNets, and we establish analogous backward-propagation results, which directly relate to the problem of training fully-connected deep ResNets. Then, we investigate the more general setting of doubly infinite NNs, where both network's width and network's depth grow unboundedly. We focus on doubly infinite fully-connected ResNets, for which we consider i.i.d. initializations. Under this setting, we show that the dynamics of quantities of interest converge, at initialization, to deterministic limits. This allow us to provide analytical expressions for inference, both in the case of weakly trained and fully trained ResNets. Our results highlight a limited expressive power of doubly infinite ResNets when the unscaled network's parameters are i.i.d. and the residual blocks are shallow.

Feed-forward chains of recurrent attractor neural networks with finite dilution near saturation

Feed-Forward Chains of Recurrent Attractor Neural Networks Near Saturation

Effect of dilution in asymmetric recurrent neural networks

Qualitative Analysis for Recurrent Neural Networks with Linear Threshold Transfer Functions

Continuous Attractors of Discrete-Time Recurrent Neural Networks

Attractor Modulation and Proliferation in 1+$\infty$ Dimensional Neural Networks

Neural networks with transient state dynamics

An Evaluation of the Dynamics of Diluted Neural Network.

Slow and Weak Attractor Computation Embedded in Fast and Strong E-I Balanced Neural Dynamics.

Analysis of Continuous Attractors for 2-D Linear Threshold Neural Networks

Cortical attractor network dynamics with diluted connectivity.

An evaluation of the dynamics of diluted neural network

Weak Signal Detection and Propagation in Diluted Feed-Forward Neural Network with Recurrent Excitation and Inhibition

Diluted neural networks with adapting and correlated synapses

Fine Discrimination of Analog Patterns by Nonlinear Dendritic Inhibition

Simultaneous embedding of multiple attractor manifolds in a recurrent neural network using constrained gradient optimization

An Analysis on Neural Dynamics with Saturated Sigmoidal Functions

Doubly infinite residual neural networks: a diffusion process approach

Statistical Neurodynamics for Sequence Processing Neural Networks with Finite Dilution

Dynamical properties of a small heterogeneous chain network of neurons in discrete time