Abstract:The Long Short-Term Memory (LSTM) recurrent neural network is capable of processing complex sequential information since it utilizes special gating schemes for learning representations from long input sequences. It has the potential to model any sequential time-series data, where the current hidden state has to be considered in the context of the past hidden states. This property makes LSTM an ideal choice to learn the complex dynamics present in long sequences. Unfortunately, the conventional LSTMs do not consider the impact of spatio-temporal dynamics corresponding to the given salient motion patterns, when they gate the information that ought to be memorized through time. To address this problem, we propose a differential gating scheme for the LSTM neural network, which emphasizes on the change in information gain caused by the salient motions between the successive video frames. This change in information gain is quantified by Derivative of States (DoS), and thus the proposed LSTM model is termed as differential Recurrent Neural Network (dRNN). In addition, the original work used the hidden state at the last time-step to model the entire video sequence. Based on the energy profiling of DoS, we further propose to employ the State Energy Profile (SEP) to search for salient dRNN states and construct more informative representations. The effectiveness of the proposed model was demonstrated by automatically recognizing human actions from the real-world 2D and 3D single-person action datasets. We point out that LSTM is a special form of dRNN. As a result, we have introduced a new family of LSTMs. Our study is one of the first works towards demonstrating the potential of learning complex time-series representations via high-order derivatives of states.

Lstm With Uniqueness Attention For Human Activity Recognition

Attention-based LSTM Network for Wearable Human Activity Recognition

MMTSA: Multimodal Temporal Segment Attention Network for Efficient Human Activity Recognition

A Semisupervised Recurrent Convolutional Attention Model for Human Activity Recognition

AttnSense: Multi-level Attention Mechanism for Multimodal Human Activity Recognition

Human Activity Recognition Using LSTM-RNN Deep Neural Network Architecture

Weakly Supervised Human Activity Recognition from Wearable Sensors by Recurrent Attention Learning

Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensors

Attention-based Convolutional Neural Network for Weakly Labeled Human Activities Recognition with Wearable Sensors

AROMA: A Deep Multi-Task Learning Based Simple and Complex Human Activity Recognition Method Using Wearable Sensors

Human Activity Recognition based on Dynamic Spatio-Temporal Relations

A Hierarchical Spatio-Temporal Model for Human Activity Recognition.

Multi-STMT: Multi-Level Network for Human Activity Recognition Based on Wearable Sensors

GRU with Dual Attentions for Sensor-Based Human Activity Recognition

Understanding and Improving Deep Neural Network for Activity Recognition

Multihead-Res-SE Residual Network with Attention for Human Activity Recognition

An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data

Differential Recurrent Neural Network and its Application for Human Activity Recognition

Enhanced Attention Tracking with Multi-Branch Network for Egocentric Activity Recognition

Semi-Supervised Adversarial Learning Using LSTM for Human Activity Recognition

A Multi-Task Deep Learning Approach for Sensor-based Human Activity Recognition and Segmentation