Abstract:With the help of convolutional neural networks (CNNs), video-based human action recognition has made significant progress. CNN features that are spatial and channel-wise can provide rich information for powerful image description. However, CNNs lack the ability to process the long-term temporal dependency of an entire video and further cannot well focus on the informative motion regions of actions. Aiming at the two problems, we propose a novel video-based action recognition framework in this paper. We first represent videos with dynamic image sequences (DISs), which effectively describe videos by modeling the local spatial-temporal dynamics and dependencies. Then a channel and spatial-temporal interest points (STIPs) attention model (CSAM) based on CNNs is proposed to focus on the discriminative channels in networks and the informative spatial motion regions of human actions. Specifically, channel attention (CA) is implemented by automatically learning channel-wise convolutional features and assigning different weights for different channels. STIPs attention (SA) is encoded by projecting the detected STIPs on frames of dynamic image sequences into the corresponding convolutional feature map space. The proposed CSAM is embedded after CNN convolutional layers to refine the feature maps, followed by global average pooling to produce effective feature representations for videos. Finally frame-level video representations are fed into an LSTM to capture the temporal dependencies and make classification. Experiments on three challenging RGB-D datasets show that our method has better performance and outperforms the state-of-the-art approaches with only depth data.

Learning Motion and Content-Dependent Features with Convolutions for Action Recognition

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Joint Multi-Scale Residual and Motion Feature Learning for Action Recognition.

Temporal Interaction and Excitation for Action Recognition

Action recognition with temporal scale-invariant deep learning framework

Learning Comprehensive Motion Representation for Action Recognition

Video Action Recognition Based on Deeper Convolution Networks with Pair-Wise Frame Motion Concatenation

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

An Efficient Motion Visual Learning Method for Video Action Recognition

Temporal Distinct Representation Learning for Action Recognition

Long-term 3D Convolutional Fusion Network for Action Recognition

Human Action Recognition Based on Selected Spatio-Temporal Features Via Bidirectional LSTM

Memory-Augmented Temporal Dynamic Learning for Action Recognition

Multi-stream with Deep Convolutional Neural Networks for Human Action Recognition in Videos

Action Recognition By Learning Deep Multi-Granular Spatio-Temporal Video Representation

Action recognition with hierarchical convolutional neural networks features and bi-directional long short-term memory model

Convolutional Networks with Channel and STIPs Attention Model for Action Recognition in Videos

Action Recognition Using Co-trained Deep Convolutional Neural Networks.

Action recognition using three dimension convolution and long short term memory

Exploring Hybrid Spatio-Temporal Convolutional Networks for Human Action Recognition.

Pooling the Convolutional Layers in Deep ConvNets for Video Action Recognition