Abstract:Human actions captured in video sequences are three-dimensional signals characterizing visual appearance and motion dynamics. To learn action patterns, existing methods adopt Convolutional and/or Recurrent Neural Networks (CNNs and RNNs). CNN based methods are effective in learning spatial appearances, but are limited in modeling long-term motion dynamics. RNNs, especially Long Short-Term Memory (LSTM), are able to learn temporal motion dynamics. However, naively applying RNNs to video sequences in a convolutional manner implicitly assumes that motions in videos are stationary across different spatial locations. This assumption is valid for short-term motions but invalid when the duration of the motion is long. In this work, we propose Lattice-LSTM (L2STM), which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity. Additionally, we introduce a novel multi-modal training procedure for training our network. Unlike traditional two-stream architectures which use RGB and optical flow information as input, our two-stream model leverages both modalities to jointly train both input gates and both forget gates in the network rather than treating the two streams as separate entities with no information about the other. We apply this end-to-end system to benchmark datasets (UCF-101 and HMDB-51) of human action recognition. Experiments show that on both datasets, our proposed method outperforms all existing ones that are based on LSTM and/or CNNs of similar model complexities.

Semi‐supervised Long Short‐term Memory for Human Action Recognition

Lstm With Uniqueness Attention For Human Activity Recognition

Lattice Long Short-Term Memory for Human Action Recognition

Sequential learning for multimodal 3D human activity recognition with Long-Short Term Memory

Graph-Temporal LSTM Networks for Skeleton-Based Action Recognition

Semi-Supervised Adversarial Learning Using LSTM for Human Activity Recognition

Human Action Recognition From Digital Videos Based on Deep Learning.

DB-LSTM: Densely-connected Bi-directional LSTM for Human Action Recognition

Learning Discriminative Spatio-temporal Representations for Semi-supervised Action Recognition

Learning Explicit Shape And Motion Evolution Maps For Skeleton-Based Human Action Recognition

Human Action Recognition Based on Hierarchical Multi-Scale Adaptive Conv-Long Short-Term Memory Network

Semi-supervised human action recognition via dual-stream cross-fusion and class-aware memory bank

Human Action Recognition Based on Selected Spatio-Temporal Features Via Bidirectional LSTM

Semi-Supervised Multiple Feature Analysis for Action Recognition

Boosted Multi-Class Semi-Supervised Learning for Human Action Recognition

Long-Short Graph Memory Network for Skeleton-based Action Recognition

Action Recognition Algorithm of Spatio-Temporal Differential LSTM Based on Feature Enhancement

Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks

Deep historical long short-term memory network for action recognition

A Semisupervised Recurrent Convolutional Attention Model for Human Activity Recognition

An active semi-supervised deep learning model for human activity recognition