Abstract:The Long Short-Term Memory (LSTM) recurrent neural network is capable of processing complex sequential information since it utilizes special gating schemes for learning representations from long input sequences. It has the potential to model any sequential time-series data, where the current hidden state has to be considered in the context of the past hidden states. This property makes LSTM an ideal choice to learn the complex dynamics present in long sequences. Unfortunately, the conventional LSTMs do not consider the impact of spatio-temporal dynamics corresponding to the given salient motion patterns, when they gate the information that ought to be memorized through time. To address this problem, we propose a differential gating scheme for the LSTM neural network, which emphasizes on the change in information gain caused by the salient motions between the successive video frames. This change in information gain is quantified by Derivative of States (DoS), and thus the proposed LSTM model is termed as differential Recurrent Neural Network (dRNN). In addition, the original work used the hidden state at the last time-step to model the entire video sequence. Based on the energy profiling of DoS, we further propose to employ the State Energy Profile (SEP) to search for salient dRNN states and construct more informative representations. The effectiveness of the proposed model was demonstrated by automatically recognizing human actions from the real-world 2D and 3D single-person action datasets. We point out that LSTM is a special form of dRNN. As a result, we have introduced a new family of LSTMs. Our study is one of the first works towards demonstrating the potential of learning complex time-series representations via high-order derivatives of states.

Action Recognition Algorithm of Spatio-Temporal Differential LSTM Based on Feature Enhancement

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

An LSTM with Differential Structure and Its Application in Action Recognition

Differential Recurrent Neural Networks for Action Recognition

Spatio-Temporal Adaptive Network with Bidirectional Temporal Difference for Action Recognition

Graph-Temporal LSTM Networks for Skeleton-Based Action Recognition

Differential Recurrent Neural Network and its Application for Human Activity Recognition

Human Action Recognition Based on Selected Spatio-Temporal Features Via Bidirectional LSTM

Recurrent Spatiotemporal Feature Learning for Action Recognition.

DB-LSTM: Densely-connected Bi-directional LSTM for Human Action Recognition

DTA: Double LSTM with Temporal-Wise Attention Network for Action Recognition

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Temporal Distinct Representation Learning for Action Recognition

A Deep Learning Network for Action Recognition Incorporating Temporal Attention Mechanism

Bi-direction Hierarchical LSTM with Spatial-Temporal Attention for Action Recognition

Action Recognition Based on Two-Stream Convolutional Networks with Long-Short-Term Spatiotemporal Features

Recurrent Attention Network Using Spatial-Temporal Relations for Action Recognition

Deep spatiotemporal LSTM network with temporal pattern feature for 3D human action recognition

Deep historical long short-term memory network for action recognition

ResLNet: Deep Residual LSTM Network with Longer Input for Action Recognition

DSTC-Net: differential spatio-temporal correlation network for similar action recognition