Abstract:Human actions captured in video sequences are three-dimensional signals characterizing visual appearance and motion dynamics. To learn action patterns, existing methods adopt Convolutional and/or Recurrent Neural Networks (CNNs and RNNs). CNN based methods are effective in learning spatial appearances, but are limited in modeling long-term motion dynamics. RNNs, especially Long Short-Term Memory (LSTM), are able to learn temporal motion dynamics. However, naively applying RNNs to video sequences in a convolutional manner implicitly assumes that motions in videos are stationary across different spatial locations. This assumption is valid for short-term motions but invalid when the duration of the motion is long. In this work, we propose Lattice-LSTM (L2STM), which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity. Additionally, we introduce a novel multi-modal training procedure for training our network. Unlike traditional two-stream architectures which use RGB and optical flow information as input, our two-stream model leverages both modalities to jointly train both input gates and both forget gates in the network rather than treating the two streams as separate entities with no information about the other. We apply this end-to-end system to benchmark datasets (UCF-101 and HMDB-51) of human action recognition. Experiments show that on both datasets, our proposed method outperforms all existing ones that are based on LSTM and/or CNNs of similar model complexities.

Human Tumble Action Recognition Using Spiking Neuron Network

A Gesture Recognition Method Based On Spiking Neural Networks For Cognition Development

Human Action Recognition Based on Spiking Neural Networks

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

2D versus 3D Convolutional Spiking Neural Networks Trained with Unsupervised STDP for Human Action Recognition

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Recognizing Human Actions by Using the Evolving Remote Supervised Method of Spiking Neural Networks.

High Efficient LSTM-based Network for Human Interaction Understanding

Human motion recognition information processing system based on LSTM Recurrent Neural Network Algorithm

An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition

Action Recognition Scheme Based on Skeleton Representation with DS-LSTM Network.

Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition

Hierarchical recurrent neural network for skeleton based action recognition

ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition

Lattice Long Short-Term Memory for Human Action Recognition

Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning

Action Recognition Based on Joint Trajectory Maps with Convolutional Neural Networks

An improved spatial temporal graph convolutional network for robust skeleton-based action recognition

Human poses recognition based on Spiking Pulse Graph Neural Networks

Human Skeleton Tree Recurrent Neural Network with Joint Relative Motion Feature for Skeleton Based Action Recognition

Skeleton-based action recognition with hierarchical spatial reasoning and temporal stack learning network