Abstract:Recurrent neural networks (RNNs) are powerful dynamical models for data with complex temporal structure. However, training RNNs has traditionally proved challenging due to exploding or vanishing of gradients. RNN models such as LSTMs and GRUs (and their variants) significantly mitigate these issues associated with training by introducing various types of gating units into the architecture. While these gates empirically improve performance, how the addition of gates influences the dynamics and trainability of GRUs and LSTMs is not well understood. Here, we take the perspective of studying randomly initialized LSTMs and GRUs as dynamical systems, and ask how the salient dynamical properties are shaped by the gates. We leverage tools from random matrix theory and mean-field theory to study the state-to-state Jacobians of GRUs and LSTMs. We show that the update gate in the GRU and the forget gate in the LSTM can lead to an accumulation of slow modes in the dynamics. Moreover, the GRU update gate can poise the system at a marginally stable point. The reset gate in the GRU and the output and input gates in the LSTM control the spectral radius of the Jacobian, and the GRU reset gate also modulates the complexity of the landscape of fixed-points. Furthermore, for the GRU we obtain a phase diagram describing the statistical properties of fixed-points. We also provide a preliminary comparison of training performance to the various dynamical regimes realized by varying hyperparameters. Looking to the future, we have introduced a powerful set of techniques which can be adapted to a broad class of RNNs, to study the influence of various architectural choices on dynamics, and potentially motivate the principled discovery of novel architectures.

Gated recurrent units viewed through the lens of continuous time dynamical systems

Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs

Gated Graph Recurrent Neural Networks

Continuous Attractors of Discrete-Time Recurrent Neural Networks

Discrete Event, Continuous Time RNNs

Recurrent attention unit: A new gated recurrent unit for long-term memory of important parts in sequential data

Back to the Continuous Attractor

Generic Properties of Random Gene Regulatory Networks.

A Homotopy Gated Recurrent Unit for Predicting High Dimensional Hyperchaos

Memristor-Based Circuit Demonstration of Gated Recurrent Unit for Predictable Neural Network

Deep Gate Recurrent Neural Network

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Autoregressive GNN-ODE GRU Model for Network Dynamics

Memory Visualization for Gated Recurrent Neural Networks in Speech Recognition

Memory-Gated Recurrent Networks

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Stimulus-Driven and Spontaneous Dynamics in Excitatory-Inhibitory Recurrent Neural Networks for Sequence Representation

Recurrent Attention Unit

The Statistical Recurrent Unit

Prototypical Recurrent Unit

Trainability, Expressivity and Interpretability in Gated Neural ODEs