Abstract:Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been dicult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades rst yielded and then made practical these powerful learning models. When appropriate, we reconcile conicting notation and nomenclature. Our goal is to provide a selfcontained explication of the state of the art together with a historical perspective and references to primary research.

Machine Learning on Sequential Data Using a Recurrent Weighted Average

Residual Recurrent Neural Networks for Learning Sequential Representations.

Efficiently applying attention to sequential data with the Recurrent Discounted Attention unit

Learning The Sequential Temporal Information with Recurrent Neural Networks

A Critical Review of Recurrent Neural Networks for Sequence Learning

Explaining Black Boxes on Sequential Data using Weighted Automata

Recurrent Attentive Neural Process for Sequential Data

A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures

Multiple-Weight Recurrent Neural Networks.

Resurrecting Recurrent Neural Networks for Long Sequences

Continual Learning with Gated Incremental Memories for sequential data processing

Decision-Guided Weighted Automata Extraction from Recurrent Neural Networks.

Recurrent Convolutional Neural Network for Sequential Recommendation.

Recurrently Controlled Recurrent Networks

Variable Computation in Recurrent Neural Networks

Understanding Hidden Memories of Recurrent Neural Networks

Riemannian metrics for neural networks II: recurrent networks and learning symbolic data sequences

Learning Useful Representations of Recurrent Neural Network Weight Matrices

Reversible Recurrent Neural Networks

Learning Longer Memory in Recurrent Neural Networks

Recurrent Attention Unit