Abstract:Memory is one of the most important mechanisms in recurrent neural networks (RNNs) learning. It plays a crucial role in practical applications, such as sequence learning. With a good memory mechanism, long term history can be fused with current information, and can thus improve RNNs learning. Developing a suitable memory mechanism is always desirable in the field of RNNs. This paper proposes a novel memory mechanism for RNNs. The main contributions of this paper are: 1) an auxiliary memory unit (AMU) is proposed, which results in a new special RNN model (AMU-RNN), separating the memory and output explicitly and 2) an efficient learning algorithm is developed by employing the technique of error flow truncation. The proposed AMU-RNN model, together with the developed learning algorithm, can learn and maintain stable memory over a long time range. This method overcomes both the learning conflict problem and gradient vanishing problem. Unlike the traditional method, which mixes the memory and output with a single neuron in a recurrent unit, the AMU provides an auxiliary memory neuron to maintain memory in particular. By separating the memory and output in a recurrent unit, the problem of learning conflicts can be eliminated easily. Moreover, by using the technique of error flow truncation, each auxiliary memory neuron ensures constant error flow during the learning process. The experiments demonstrate good performance of the proposed AMU-RNNs and the developed learning algorithm. The method exhibits quite efficient learning performance with stable convergence in the AMU-RNN learning and outperforms the state-of-the-art RNN models in sequence generation and sequence classification tasks.

Recurrent Neural Networks with External Addressable Long-Term and Working Memory for Learning Long-Term Dependences.

On extended long short-term memory and dependent bidirectional recurrent neural network

Residual Recurrent Neural Networks for Learning Sequential Representations.

RECURRENT NEURAL NETWORK BASED LANGUAGE MODELING WITH CONTROLLABLE EXTERNAL MEMORY

Spoken Language Understanding Method Based on Recurrent Neural Network with Persistent Memory

Linguistic Knowledge as Memory for Recurrent Neural Networks.

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Nonrecurrent Neural Structure for Long-Term Dependence.

PRNN: Recurrent Neural Network with Persistent Memory.

Relational recurrent neural networks

Residual Memory Networks: Feed-forward approach to learn long temporal dependencies

Learning a World Model With Multitimescale Memory Augmentation

Memory-Gated Recurrent Networks

Learning various length dependence by dual recurrent neural networks

Recurrent Neural Networks with Auxiliary Memory Units

Learning Longer Memory in Recurrent Neural Networks

RecallM: An Adaptable Memory Mechanism with Temporal Understanding for Large Language Models

Gating Recurrent Enhanced Memory Neural Networks on Language Identification.

Learning to Update Auto-associative Memory in Recurrent Neural Networks for Improving Sequence Memorization.

Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate

Ordered Memory.