Abstract:Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mechanism concept can help RNNs remember and forget previous information. Representing the hidden layers of an RNN with more expressive operations (i.e., tensor products) helps it learn a more complex relationship between the current input and the previous hidden layer information. These ideas can generally improve RNN performances. In this paper, we proposed a novel RNN architecture that combine the concepts of gating mechanism and the tensor product into a single model. By combining these two concepts into a single RNN, our proposed models learn long-term dependencies by modeling with gating units and obtain more expressive and direct interaction between input and hidden layers using a tensor product on 3-dimensional array (tensor) weight parameters. We use Long Short Term Memory (LSTM) RNN and Gated Recurrent Unit (GRU) RNN and combine them with a tensor product inside their formulations. Our proposed RNNs, which are called a Long-Short Term Memory Recurrent Neural Tensor Network (LSTMRNTN) and Gated Recurrent Unit Recurrent Neural Tensor Network (GRURNTN), are made by combining the LSTM and GRU RNN models with the tensor product. We conducted experiments with our proposed models on word-level and character-level language modeling tasks and revealed that our proposed models significantly improved their performance compared to our baseline models.

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Residual Recurrent Neural Networks for Learning Sequential Representations.

HGRN2: Gated Linear RNNs with State Expansion

Gated Recurrent Neural Tensor Network

Highway State Gating for Recurrent Highway Networks: improving information flow through time

Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units

Non-local Recurrent Neural Memory for Supervised Sequence Modeling

GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

Gated Graph Recurrent Neural Networks

Memory-Gated Recurrent Networks

Investigating Gated Recurrent Neural Networks for Acoustic Modeling

Gates Are Not What You Need in RNNs

Minimal Gated Unit for Recurrent Neural Networks

Sentence Modeling with Gated Recursive Neural Network

Introducing the Hidden Neural Markov Chain framework

Sliding Hierarchical Recurrent Neural Networks for Sequence Classification

Deep Gate Recurrent Neural Network