Abstract:Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been dicult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades rst yielded and then made practical these powerful learning models. When appropriate, we reconcile conicting notation and nomenclature. Our goal is to provide a selfcontained explication of the state of the art together with a historical perspective and references to primary research.

Research Progress of RNN Language Model

Residual Recurrent Neural Networks for Learning Sequential Representations.

Recurrent Memory Networks for Language Modeling

A Study on Neural Network Language Modeling

Progress in Neural NLP: Modeling, Learning, and Reasoning

Research and Progress in the Language Network

A Novel Natural Language Processing Model in Mobile Communication Networks

An Improved Recurrent Neural Network Language Model for Programming Language

Exploring the Limits of Language Modeling

Recurrent Neural Network Based Language Model Adaptation for Accent Mandarin Speech.

Improving Accented Mandarin Speech Recognition by Using Recurrent Neural Network Based Language Model Adaptation

Comparative Study of CNN and RNN for Natural Language Processing

Recurrent Neural Networks (RNNs): A gentle Introduction and Overview

History, Development, and Principles of Large Language Models-An Introductory Survey

NEWLSTM: an Optimized Long Short-Term Memory Language Model for Sequence Prediction.

Recurrent Neural Network Language Model with Part-of-speech for Mandarin Speech Recognition.

Computational Models to Study Language Processing in the Human Brain: A Survey

A Critical Review of Recurrent Neural Networks for Sequence Learning

Analyzing Linguistic Knowledge in Sequential Model of Sentence.

Evolution and advancements in deep learning models for Natural Language Processing

DRRNets: Dynamic Recurrent Routing Via Low-Rank Regularization in Recurrent Neural Networks.