Abstract:With the help of convolutional neural networks (CNNs), video-based human action recognition has made significant progress. CNN features that are spatial and channel-wise can provide rich information for powerful image description. However, CNNs lack the ability to process the long-term temporal dependency of an entire video and further cannot well focus on the informative motion regions of actions. Aiming at the two problems, we propose a novel video-based action recognition framework in this paper. We first represent videos with dynamic image sequences (DISs), which effectively describe videos by modeling the local spatial-temporal dynamics and dependencies. Then a channel and spatial-temporal interest points (STIPs) attention model (CSAM) based on CNNs is proposed to focus on the discriminative channels in networks and the informative spatial motion regions of human actions. Specifically, channel attention (CA) is implemented by automatically learning channel-wise convolutional features and assigning different weights for different channels. STIPs attention (SA) is encoded by projecting the detected STIPs on frames of dynamic image sequences into the corresponding convolutional feature map space. The proposed CSAM is embedded after CNN convolutional layers to refine the feature maps, followed by global average pooling to produce effective feature representations for videos. Finally frame-level video representations are fed into an LSTM to capture the temporal dependencies and make classification. Experiments on three challenging RGB-D datasets show that our method has better performance and outperforms the state-of-the-art approaches with only depth data.

Human Action Recognition Based on Improved Fusion Attention CNN and RNN

Residual Attention Fusion Network for Video Action Recognition

Bi-direction Hierarchical LSTM with Spatial-Temporal Attention for Action Recognition

Human-Object Contour for Action Recognition with Attentional Multi-modal Fusion Network

Multiple Feature Fusion in Convolutional Neural Networks for Action Recognition

Human Action Recognition in Videos using Convolution Long Short-Term Memory Network with Spatio-Temporal Networks

Attend It Again: Recurrent Attention Convolutional Neural Network for Action Recognition

3D-Cnn-based Fused Feature Maps with LSTM Applied to Action Recognition.

Recurrent Attention Network Using Spatial-Temporal Relations for Action Recognition

Joint Network based Attention for Action Recognition

Action Recognition by an Attention-Aware Temporal Weighted Convolutional Neural Network.

Two Stream LSTM: A Deep Fusion Framework for Human Action Recognition

Action Recognition with Joint Attention on Multi-Level Deep Features

Convolutional Networks with Channel and STIPs Attention Model for Action Recognition in Videos

Attention-Driven Appearance-Motion Fusion Network for Action Recognition.

Action Recognition in Videos with Temporal Segments Fusions

DTA: Double LSTM with Temporal-Wise Attention Network for Action Recognition

Fully Convolutional Networks for Action Recognition

Integrating Temporal and Spatial Attention for Video Action Recognition

Human Action Recognition Combining Sequential Dynamic Images and Two-Stream Convolutional Network

Action Recognition Based on Two-Stream Convolutional Networks with Long-Short-Term Spatiotemporal Features