Abstract:Mathematical methods are developed to characterize the asymptotics of recurrent neural networks (RNN) as the number of hidden units, data samples in the sequence, hidden state updates, and training steps simultaneously grow to infinity. In the case of an RNN with a simplified weight matrix, we prove the convergence of the RNN to the solution of an infinite-dimensional ODE coupled with the fixed point of a random algebraic equation. The analysis requires addressing several challenges which are unique to RNNs. In typical mean-field applications (e.g., feedforward neural networks), discrete updates are of magnitude $\mathcal{O}(\frac{1}{N})$ and the number of updates is $\mathcal{O}(N)$. Therefore, the system can be represented as an Euler approximation of an appropriate ODE/PDE, which it will converge to as $N \rightarrow \infty$. However, the RNN hidden layer updates are $\mathcal{O}(1)$. Therefore, RNNs cannot be represented as a discretization of an ODE/PDE and standard mean-field techniques cannot be applied. Instead, we develop a fixed point analysis for the evolution of the RNN memory states, with convergence estimates in terms of the number of update steps and the number of hidden units. The RNN hidden layer is studied as a function in a Sobolev space, whose evolution is governed by the data sequence (a Markov chain), the parameter updates, and its dependence on the RNN hidden layer at the previous time step. Due to the strong correlation between updates, a Poisson equation must be used to bound the fluctuations of the RNN around its limit equation. These mathematical methods give rise to the neural tangent kernel (NTK) limits for RNNs trained on data sequences as the number of data samples and size of the neural network grow to infinity.

On the rate of convergence of a deep recurrent neural network estimate in a regression problem with dependent data

Analysis of the rate of convergence of fully connected deep neural network regression estimates with smooth activation function

Statistical Properties of Deep Neural Networks with Dependent Data

Estimation of a regression function on a manifold by fully connected deep neural networks

On the universal consistency of an over-parametrized deep neural network estimate learned by gradient descent

Nonparametric regression using deep neural networks with ReLU activation function

Asymptotics of Linear Regression with Linearly Dependent Data

Regression from Dependent Observations

Asymptotic Convergence Rate of Dropout on Shallow Linear Neural Networks

Dimension-independent rates for structured neural density estimation

Analysis of the expected $L_2$ error of an over-parametrized deep neural network estimate learned by gradient descent without regularization

Convergence rates of non-stationary and deep Gaussian process regression

Nonparametric regression for repeated measurements with deep neural networks

Sparse-penalized deep neural networks estimator under weak dependence

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

Theoretical analysis of deep neural networks for temporally dependent observations

Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimension

Over-parameterized Deep Nonparametric Regression for Dependent Data with Its Applications to Reinforcement Learning.

Kernel Limit of Recurrent Neural Networks Trained on Ergodic Data Sequences

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Robust deep learning from weakly dependent data