Abstract:The key challenge of sequence representation learning is to capture the long-range temporal dependencies. Typical methods for supervised sequence representation learning are built upon recurrent neural networks to capture temporal dependencies. One potential limitation of these methods is that they only model one-order information interactions explicitly between adjacent time steps in a sequence, hence the high-order interactions between nonadjacent time steps are not fully exploited. It greatly limits the capability of modeling the long-range temporal dependencies since the temporal features learned by one-order interactions cannot be maintained for a long term due to temporal information dilution and gradient vanishing. To tackle this limitation, we propose the non-local recurrent neural memory ( NRNM ) for supervised sequence representation learning, which performs non-local operations by means of self-attention mechanism to learn full-order interactions within a sliding temporal memory block and models global interactions between memory blocks in a gated recurrent manner. Consequently, our model is able to capture long-range dependencies. Besides, the latent high-level features contained in high-order interactions can be distilled by our model. We validate the effectiveness and generalization of our NRNM on three types of sequence applications across different modalities, including sequence classification, step-wise sequential prediction and sequence similarity learning. Our model compares favorably against other state-of-the-art methods specifically designed for each of these sequence applications.

Spontaneous Temporal Grouping Neural Network for Long-Term Memory Modeling

A hippocampus CA3 spiking neural network model for storage and retrieval of sequential memory

A Hierarchical Organized Memory Model Using Spiking Neurons

A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

An Optimization Scheme for Segmented-Memory Neural Network

Unleashing the Potential of Spiking Neural Networks for Sequential Modeling with Contextual Embedding.

Temporal-Sequential Learning with Columnar-Structured Spiking Neural Networks.

A Spiking Neural Network Model For Associative Memory Using Temporal Codes

Nonrecurrent Neural Structure for Long-Term Dependence.

How the Brain Formulates Memory: A Spatio-Temporal Model Research Frontier.

Non-local Recurrent Neural Memory for Supervised Sequence Modeling

A New Hierarchical Temporal Memory Based on Recurrent Learning Unit

Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency.

Associative Memory and Segmentation in the Network Composed of Spiking Neurons

Tree Memory Networks for Modelling Long-term Temporal Dependencies

Spatio-Temporal Sequential Memory Model with Mini-Column Neural Network

Learning Longer Memory in Recurrent Neural Networks

On extended long short-term memory and dependent bidirectional recurrent neural network

Memory-Gated Recurrent Networks

Learning Sequence Representations by Non-local Recurrent Neural Memory

Lifelong Learning Process: Self-Memory Supervising and Dynamically Growing Networks