Abstract:Mathematical methods are developed to characterize the asymptotics of recurrent neural networks (RNN) as the number of hidden units, data samples in the sequence, hidden state updates, and training steps simultaneously grow to infinity. In the case of an RNN with a simplified weight matrix, we prove the convergence of the RNN to the solution of an infinite-dimensional ODE coupled with the fixed point of a random algebraic equation. The analysis requires addressing several challenges which are unique to RNNs. In typical mean-field applications (e.g., feedforward neural networks), discrete updates are of magnitude $\mathcal{O}(\frac{1}{N})$ and the number of updates is $\mathcal{O}(N)$. Therefore, the system can be represented as an Euler approximation of an appropriate ODE/PDE, which it will converge to as $N \rightarrow \infty$. However, the RNN hidden layer updates are $\mathcal{O}(1)$. Therefore, RNNs cannot be represented as a discretization of an ODE/PDE and standard mean-field techniques cannot be applied. Instead, we develop a fixed point analysis for the evolution of the RNN memory states, with convergence estimates in terms of the number of update steps and the number of hidden units. The RNN hidden layer is studied as a function in a Sobolev space, whose evolution is governed by the data sequence (a Markov chain), the parameter updates, and its dependence on the RNN hidden layer at the previous time step. Due to the strong correlation between updates, a Poisson equation must be used to bound the fluctuations of the RNN around its limit equation. These mathematical methods give rise to the neural tangent kernel (NTK) limits for RNNs trained on data sequences as the number of data samples and size of the neural network grow to infinity.

Finite Size Effects in Separable Recurrent Neural Networks

Stochastic Neural Networks with Infinite Width are Deterministic

Finite size effects in the correlation structure of stochastic neural networks: analysis of different connectivity matrices and failure of the mean-field theory

Extracting Critical Exponents by Finite-Size Scaling with Convolutional Neural Networks

Finite-size analysis in neural network classification of critical phenomena

Kernel Limit of Recurrent Neural Networks Trained on Ergodic Data Sequences

Finite size scaling for homogeneous pinning models

A stochastic-field description of finite-size spiking neural networks

Finite size effects in non-equilibrium membrane phase separation

Finite-size scaling in the canonical ensemble

Large deviations, dynamics and phase transitions in large stochastic heterogeneous neural networks

Mathematical studies of the dynamics of finite-size binary neural networks: A review of recent progress

Finite-size effects on current correlation functions

Finite-size and finite-time effects in large deviation functions near dynamical symmetry breaking transitions

Scaling ResNets in the Large-depth Regime

Finite Size Scaling of Perceptron

Random recurrent neural networks with delays

Finite-size scaling of geometric renormalization flows in complex networks

Discontinuous transition to chaos in a canonical random neural network

Understanding finite size effects in quasi-long-range orders for exactly solvable chain models

Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks