Abstract:Human actions captured in video sequences are three-dimensional signals characterizing visual appearance and motion dynamics. To learn action patterns, existing methods adopt Convolutional and/or Recurrent Neural Networks (CNNs and RNNs). CNN based methods are effective in learning spatial appearances, but are limited in modeling long-term motion dynamics. RNNs, especially Long Short-Term Memory (LSTM), are able to learn temporal motion dynamics. However, naively applying RNNs to video sequences in a convolutional manner implicitly assumes that motions in videos are stationary across different spatial locations. This assumption is valid for short-term motions but invalid when the duration of the motion is long. In this work, we propose Lattice-LSTM (L2STM), which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity. Additionally, we introduce a novel multi-modal training procedure for training our network. Unlike traditional two-stream architectures which use RGB and optical flow information as input, our two-stream model leverages both modalities to jointly train both input gates and both forget gates in the network rather than treating the two streams as separate entities with no information about the other. We apply this end-to-end system to benchmark datasets (UCF-101 and HMDB-51) of human action recognition. Experiments show that on both datasets, our proposed method outperforms all existing ones that are based on LSTM and/or CNNs of similar model complexities.

Recognition of Visually Perceived Compositional Human Actions by Multiple Spatio-Temporal Scales Recurrent Neural Networks

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Graph-Temporal LSTM Networks for Skeleton-Based Action Recognition

Multi-Level Recurrent Residual Networks for Action Recognition

Multi-Branch Spatial-Temporal Network for Action Recognition

Human Action Recognition Based on Three-Stream Network with Frame Sequence Features

Human Action Recognition Based on Hierarchical Multi-Scale Adaptive Conv-Long Short-Term Memory Network

Predictive Coding for Dynamic Vision : Development of Functional Hierarchy in a Multiple Spatio-Temporal Scales RNN Model

Multiple Timescale Recurrent Neural Network with Slow Feature Analysis for Efficient Motion Recognition.

Spatial-Temporal Neural Networks For Action Recognition

I3D-LSTM: A New Model for Human Action Recognition

Spatial-Temporal Hypergraph Neural Network based on Attention Mechanism for Multi-view Data Action Recognition

Human Action Recognition by Learning Spatio-Temporal Features with Deep Neural Networks

Spatiotemporal Multi-Task Network for Human Activity Understanding.

Supervised Multiple Timescale Recurrent Neuron Network Model for Human Action Classification.

A Human Action Recognition Model Inspired By Multiple Scale Temporal Segments Model Fusion

Bi-direction Hierarchical LSTM with Spatial-Temporal Attention for Action Recognition

Multi‐mode Neural Network for Human Action Recognition

Multi-View Time-Series Hypergraph Neural Network for Action Recognition

Hierarchical Multi-scale Attention Networks for Action Recognition

Lattice Long Short-Term Memory for Human Action Recognition