Abstract:Recently, deep recurrent neural networks (DRNNs) have been widely proposed for language modeling. DRNNs can learn higher-level features of input data by stacking multiple recurrent layers, making them achieve better performance than single-layer models. However, due to their simple linear stacking patterns, the gradient information vanishes when it is backward propagated through too many layers. As a result, DRNNs become hard to train and their performance degrades rapidly with the number of recurrent layers increasing. To address this problem, the feature memory-based deep recurrent neural network (FMDRNN) is proposed in this paper. FMDRNN presents a new stacking pattern by introducing a special feature memory module (FM), which makes the hidden units of each layer can see and reuse all the features generated by preceding stacked layers, not just the feature from previous layer as in DRNNs. FM is like a traffic hub to provide direct connections between each two layers, and the attention network in FM controls the switch of these connections. These direct connections enable FMDRNN can alleviate the vanishing of gradient in the process of backward propagation and also make the learned features do not wash away when they reach the end of the network. FMDRNN is evaluated by performing extensive experiments on the widely used English Penn Treebank dataset and five more complex non-English language corpora. The experimental results show that FMDRNN can be effectively trained even if a larger number of layers are stacked, so that it benefits from deeper networks instead of degrading performance, and consistently achieves markedly better results than other models through deeper but thinner network. (C) 2018 Elsevier B.V. All rights reserved.

RECURRENT NEURAL NETWORK BASED LANGUAGE MODELING WITH CONTROLLABLE EXTERNAL MEMORY

Attention-based Memory Selection Recurrent Network for Language Modeling

Recurrent Memory Networks for Language Modeling

Recurrent Neural Networks with External Addressable Long-Term and Working Memory for Learning Long-Term Dependences.

On extended long short-term memory and dependent bidirectional recurrent neural network

Spoken Language Understanding Method Based on Recurrent Neural Network with Persistent Memory

Augmenting Language Models with Long-Term Memory

Extending Memory for Language Modelling

MEMORYLLM: Towards Self-Updatable Large Language Models

NEWLSTM: an Optimized Long Short-Term Memory Language Model for Sequence Prediction.

Large Language Models Are Semi-Parametric Reinforcement Learning Agents

Feature Memory-Based Deep Recurrent Neural Network for Language Modeling

Learning Efficient Lexically-Constrained Neural Machine Translation with External Memory

ELSTM: An improved long short‐term memory network language model for sequence learning

Nonrecurrent Neural Structure for Long-Term Dependence.

Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System.

LaMemo: Language Modeling with Look-Ahead Memory

Generic Memory Modeling with Recurrent Neural Network

Recurrent Neural Network Based Language Model Adaptation for Accent Mandarin Speech.

Gating Recurrent Enhanced Memory Neural Networks on Language Identification.