Abstract:Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been dicult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades rst yielded and then made practical these powerful learning models. When appropriate, we reconcile conicting notation and nomenclature. Our goal is to provide a selfcontained explication of the state of the art together with a historical perspective and references to primary research.

Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

Residual Recurrent Neural Networks for Learning Sequential Representations.

Learning Sequence Representations by Non-local Recurrent Neural Memory

Riemannian metrics for neural networks II: recurrent networks and learning symbolic data sequences

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Recurrently Controlled Recurrent Networks

The Power of Linear Recurrent Neural Networks

A Critical Review of Recurrent Neural Networks for Sequence Learning

Stimulus-Driven and Spontaneous Dynamics in Excitatory-Inhibitory Recurrent Neural Networks for Sequence Representation

RotRNN: Modelling Long Sequences with Rotations

Representation of linguistic form and function in recurrent neural networks

Generating Sequences With Recurrent Neural Networks

Colorless green recurrent networks dream hierarchically

Learning The Sequential Temporal Information with Recurrent Neural Networks

Recurrent Network Models Of Sequence Generation And Memory

Reversible Recurrent Neural Networks

Encoding Sensory and Motor Patterns as Time-Invariant Trajectories in Recurrent Neural Networks

Learning Longer Memory in Recurrent Neural Networks

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks

On Efficiently Representing Regular Languages as RNNs