Abstract:The advantage of recurrent neural networks (RNNs) in learning dependencies between time-series data has distinguished RNNs from other deep learning models. Recently, many advances are proposed in this emerging field. However, there is a lack of comprehensive review on memory models in RNNs in the literature. This paper provides a fundamental review on RNNs and long short term memory (LSTM) model. Then, provides a surveys of recent advances in different memory enhancements and learning techniques for capturing long term dependencies in RNNs.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is related to the challenges encountered by Recurrent Neural Networks (RNNs) in learning long - time - series data dependencies, especially how the Long - Short - Term Memory (LSTM) model can be improved to better capture long - time dependencies. Specifically, the paper focuses on the following aspects: 1. **Long - time - dependency problem**: The main problems that RNNs face when processing long - time - series data are vanishing gradients and exploding gradients, which limit the model's ability to learn long - term dependencies. The paper explores how the LSTM model and its variants can alleviate these problems by introducing gating mechanisms, thereby enhancing the ability to learn long - time dependencies. 2. **Model overview and comparison**: The paper provides a comprehensive review of RNNs and LSTM models, including their basic structures and working principles. In addition, it also surveys the progress in enhancing memory capabilities and learning techniques in recent years, such as stacked LSTM, bidirectional LSTM, multi - dimensional LSTM, grid LSTM, etc. 3. **Performance improvement**: The paper discusses how to improve the performance of the LSTM model through different methods and techniques, for example, by introducing a forget gate to control the information decay rate in the memory unit, and by using multi - dimensional and grid structures to handle multi - dimensional data. 4. **Application scenarios**: The paper also mentions the applications of the LSTM model in multiple fields, such as handwriting recognition, speech recognition, speech generation, sequence - to - sequence mapping, etc., demonstrating the effectiveness and potential of LSTM in practical problems. In summary, this paper aims to comprehensively review and analyze the research progress of the LSTM model in learning long - time dependencies and explore possible future research directions.

Learning Over Long Time Lags

Residual Recurrent Neural Networks for Learning Sequential Representations.

A survey on long short-term memory networks for time series prediction

Learning Longer Memory in Recurrent Neural Networks

Residual Memory Networks: Feed-forward approach to learn long temporal dependencies

Do RNN and LSTM have Long Memory?

On the Long-Term Memory of Deep Recurrent Networks

Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey

A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures

Learning various length dependence by dual recurrent neural networks

On extended long short-term memory and dependent bidirectional recurrent neural network

A review on the long short-term memory model

Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency.

A Performance Review of Recurrent Neural Networks Long Short-Term Memory (LSTM)

Learning Long-Term Dependencies in Irregularly-Sampled Time Series

A Critical Review of Recurrent Neural Networks for Sequence Learning

Tree Memory Networks for Modelling Long-term Temporal Dependencies

LSTM Inefficiency in Long-Term Dependencies Regression Problems

Nonrecurrent Neural Structure for Long-Term Dependence.

Can recurrent neural networks learn process model structure?

Subtraction Gates: Another Way to Learn Long-Term Dependencies in Recurrent Neural Networks