Abstract:Recurrent neural networks (RNNs) are powerful dynamical models for data with complex temporal structure. However, training RNNs has traditionally proved challenging due to exploding or vanishing of gradients. RNN models such as LSTMs and GRUs (and their variants) significantly mitigate these issues associated with training by introducing various types of gating units into the architecture. While these gates empirically improve performance, how the addition of gates influences the dynamics and trainability of GRUs and LSTMs is not well understood. Here, we take the perspective of studying randomly initialized LSTMs and GRUs as dynamical systems, and ask how the salient dynamical properties are shaped by the gates. We leverage tools from random matrix theory and mean-field theory to study the state-to-state Jacobians of GRUs and LSTMs. We show that the update gate in the GRU and the forget gate in the LSTM can lead to an accumulation of slow modes in the dynamics. Moreover, the GRU update gate can poise the system at a marginally stable point. The reset gate in the GRU and the output and input gates in the LSTM control the spectral radius of the Jacobian, and the GRU reset gate also modulates the complexity of the landscape of fixed-points. Furthermore, for the GRU we obtain a phase diagram describing the statistical properties of fixed-points. We also provide a preliminary comparison of training performance to the various dynamical regimes realized by varying hyperparameters. Looking to the future, we have introduced a powerful set of techniques which can be adapted to a broad class of RNNs, to study the influence of various architectural choices on dynamics, and potentially motivate the principled discovery of novel architectures.

Xigmoid: an Approach to Improve the Gating Mechanism of RNN

Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units

Occam's Gates

Residual Recurrent Neural Networks for Learning Sequential Representations.

Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs

Gates Are Not What You Need in RNNs

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

Towards Binary-Valued Gates for Robust LSTM Training

Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions

Fast Training and Model Compression of Gated RNNs via Singular Value Decomposition

Deep Gate Recurrent Neural Network

Understanding Gating Operations in Recurrent Neural Networks Through Opinion Expression Extraction.

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Investigating gated recurrent neural networks for speech synthesis

Highway State Gating for Recurrent Highway Networks: improving information flow through time

HGRN2: Gated Linear RNNs with State Expansion

Dynamic Dual Gating Neural Networks

Improving speech recognition by revising gated recurrent units

An Improved Time Feedforward Connections Recurrent Neural Networks

Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformation

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts