Abstract:Recently, deep recurrent neural networks (DRNNs) have been widely proposed for language modeling. DRNNs can learn higher-level features of input data by stacking multiple recurrent layers, making them achieve better performance than single-layer models. However, due to their simple linear stacking patterns, the gradient information vanishes when it is backward propagated through too many layers. As a result, DRNNs become hard to train and their performance degrades rapidly with the number of recurrent layers increasing. To address this problem, the feature memory-based deep recurrent neural network (FMDRNN) is proposed in this paper. FMDRNN presents a new stacking pattern by introducing a special feature memory module (FM), which makes the hidden units of each layer can see and reuse all the features generated by preceding stacked layers, not just the feature from previous layer as in DRNNs. FM is like a traffic hub to provide direct connections between each two layers, and the attention network in FM controls the switch of these connections. These direct connections enable FMDRNN can alleviate the vanishing of gradient in the process of backward propagation and also make the learned features do not wash away when they reach the end of the network. FMDRNN is evaluated by performing extensive experiments on the widely used English Penn Treebank dataset and five more complex non-English language corpora. The experimental results show that FMDRNN can be effectively trained even if a larger number of layers are stacked, so that it benefits from deeper networks instead of degrading performance, and consistently achieves markedly better results than other models through deeper but thinner network. (C) 2018 Elsevier B.V. All rights reserved.

Feedforward Sequential Memory Neural Networks without Recurrent Feedback

Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency.

Nonrecurrent Neural Structure for Long-Term Dependence.

Residual Recurrent Neural Networks for Learning Sequential Representations.

Compact Feedforward Sequential Memory Networks for Large Vocabulary Continuous Speech Recognition

Deep-FSMN for Large Vocabulary Continuous Speech Recognition

Feedforward Sequential Memory Networks Based Encoder-Decoder Model for Machine Translation

Non-local Recurrent Neural Memory for Supervised Sequence Modeling

Residual Memory Networks: Feed-forward approach to learn long temporal dependencies

Attention-based Memory Selection Recurrent Network for Language Modeling

Learning Sequence Representations by Non-local Recurrent Neural Memory

Augmenting Language Models with Long-Term Memory

Research on Mongolian Speech Recognition Based on FSMN.

Feature Memory-Based Deep Recurrent Neural Network for Language Modeling

A hippocampus CA3 spiking neural network model for storage and retrieval of sequential memory

Spontaneous Temporal Grouping Neural Network for Long-Term Memory Modeling

Unleashing the Potential of Spiking Neural Networks for Sequential Modeling with Contextual Embedding.

Recurrent Neural Networks with Finite Memory Length

RECURRENT NEURAL NETWORK BASED LANGUAGE MODELING WITH CONTROLLABLE EXTERNAL MEMORY

Recurrent Memory Networks for Language Modeling

On extended long short-term memory and dependent bidirectional recurrent neural network