Abstract:The state-of-the-arts in action recognition are suffering from three challenges: (1) How to model spatial transformations of action since it is always geometric variation over time in videos. (2) How to develop the semantic action-aware temporal features from one video with a large proportion of irrelevant frames to the labeled action class, which hurt the final performance. (3) The action recognition speed of most existing models is too slow to be applied to actual scenes. In this paper, to address these three challenges, we propose a novel CNN-based action recognition method called SAST including three important modules, which can effectively learn semantic action-aware spatial-temporal features with a faster speed. Firstly, to learn action-aware spatial features (spatial transformations), we design a weight shared 2D Deformable Convolutional network named 2DDC with deformable convolutions whose receptive fields can be adaptively adjusted according to the complex geometric structure of actions. Then, we propose a light Temporal Attention model called TA to develop the action-aware temporal features that are discriminative for the labeled action category. Finally, we apply an effective 3D network to learn the temporal context between frames for building the final video-level representation. To improve the efficiency, we only utilize the raw RGB rather than optical flow and RGB as the input to our model. Experimental results on four challenging video recognition datasets Kinetics-400, Something-Something-V1, UCF101 and HMDB51 demonstrate that our proposed method can not only achieve comparable performances but be 10x to 50x faster than most of state-of-the-art action recognition methods.

Recurrent Temporal Sparse Autoencoder for Attention-Based Action Recognition.

End-to-end Temporal Attention Extraction and Human Action Recognition

ARCH: Adaptive Recurrent-Convolutional Hybrid Networks for Long-Term Action Recognition

Short-Term Action Recognition by 3D Convolutional Neural Network with Pixel-Wise Evidences

Recurrent Attention Network Using Spatial-Temporal Relations for Action Recognition

Bi-direction Hierarchical LSTM with Spatial-Temporal Attention for Action Recognition

Temporal Distinct Representation Learning for Action Recognition

Action Recognition with Bootstrapping Based Long-range Temporal Context Attention

Action Recognition Using Visual Attention with Reinforcement Learning.

Temporal Sparse Feature Auto-Combination Deep Network for Video Action Recognition.

Action recognition using attention-based spatio-temporal VLAD networks and adaptive video sequences optimization

Integrating Temporal and Spatial Attention for Video Action Recognition

EAR: Efficient Action Recognition with Local-Global Temporal Aggregation

Efficient spatiotemporal context modeling for action recognition

Learning Attention-Enhanced Spatiotemporal Representation for Action Recognition

Temporal Spiking Recurrent Neural Network for Action Recognition.

Sparse Feature Auto-Combination Deep Network for Video Action Recognition

SSTA-Net: Self-supervised Spatio-Temporal Attention Network for Action Recognition.

Efficient Spatialtemporal Context Modeling for Action Recognition

Action recognition with temporal scale-invariant deep learning framework

SAST: Learning Semantic Action-Aware Spatial-Temporal Features for Efficient Action Recognition