Abstract:Modern neural networks (NN) featuring a large number of layers (depth) and units per layer (width) have achieved a remarkable performance across many domains. While there exists a vast literature on the interplay between infinitely wide NNs and Gaussian processes, a little is known about analogous interplays with respect to infinitely deep NNs. NNs with independent and identically distributed (i.i.d.) initializations exhibit undesirable forward and backward propagation properties as the number of layers increases. To overcome these drawbacks, Peluchetti and Favaro (2020) considered fully-connected residual networks (ResNets) with network's parameters initialized by means of distributions that shrink as the number of layers increases, thus establishing an interplay between infinitely deep ResNets and solutions to stochastic differential equations, i.e. diffusion processes, and showing that infinitely deep ResNets does not suffer from undesirable forward-propagation properties. In this paper, we review the results of Peluchetti and Favaro (2020), extending them to convolutional ResNets, and we establish analogous backward-propagation results, which directly relate to the problem of training fully-connected deep ResNets. Then, we investigate the more general setting of doubly infinite NNs, where both network's width and network's depth grow unboundedly. We focus on doubly infinite fully-connected ResNets, for which we consider i.i.d. initializations. Under this setting, we show that the dynamics of quantities of interest converge, at initialization, to deterministic limits. This allow us to provide analytical expressions for inference, both in the case of weakly trained and fully trained ResNets. Our results highlight a limited expressive power of doubly infinite ResNets when the unscaled network's parameters are i.i.d. and the residual blocks are shallow.

Exploring epoch-dependent stochastic residual networks

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Universal Approximation Property of Stochastic Configuration Networks for Time Series

Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families

Deep Networks with Stochastic Depth

Stimulative Training++: Go Beyond The Performance Limits of Residual Networks

Training of deep residual networks with stochastic MG/OPT

Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes

Deeper Insights into Learning Performance of Stochastic Configuration Networks

Partially Stochastic Infinitely Deep Bayesian Neural Networks

Stochastic reserving with a stacked model based on a hybridized Artificial Neural Network

Recurrent Stochastic Configuration Networks for Temporal Data Analytics

Understanding Stochastic Optimization Behavior at the Layer Update Level (Student Abstract)

Doubly infinite residual neural networks: a diffusion process approach

Deep Recurrent Stochastic Configuration Networks for Modelling Nonlinear Dynamic Systems

Residual Connections Encourage Iterative Inference

Understanding and Exploring the Network with Stochastic Architectures.

Stochastic Recurrent Neural Network for Multistep Time Series Forecasting

Exploration-exploitation mechanisms in recurrent neural networks and human learners in restless bandit problems

Residual Switching Network for Portfolio Optimization

Stochastic Configuration Networks: Fundamentals and Algorithms