Abstract:The state-of-the-arts in action recognition are suffering from three challenges: (1) How to model spatial transformations of action since it is always geometric variation over time in videos. (2) How to develop the semantic action-aware temporal features from one video with a large proportion of irrelevant frames to the labeled action class, which hurt the final performance. (3) The action recognition speed of most existing models is too slow to be applied to actual scenes. In this paper, to address these three challenges, we propose a novel CNN-based action recognition method called SAST including three important modules, which can effectively learn semantic action-aware spatial-temporal features with a faster speed. Firstly, to learn action-aware spatial features (spatial transformations), we design a weight shared 2D Deformable Convolutional network named 2DDC with deformable convolutions whose receptive fields can be adaptively adjusted according to the complex geometric structure of actions. Then, we propose a light Temporal Attention model called TA to develop the action-aware temporal features that are discriminative for the labeled action category. Finally, we apply an effective 3D network to learn the temporal context between frames for building the final video-level representation. To improve the efficiency, we only utilize the raw RGB rather than optical flow and RGB as the input to our model. Experimental results on four challenging video recognition datasets Kinetics-400, Something-Something-V1, UCF101 and HMDB51 demonstrate that our proposed method can not only achieve comparable performances but be 10x to 50x faster than most of state-of-the-art action recognition methods.

Spatio-Temporal Collaborative Module for Efficient Action Recognition

A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Spatio-temporal Collaborative Convolution for Video Action Recognition

STSM: Spatio-Temporal Shift Module for Efficient Action Recognition

Temporal Interaction and Excitation for Action Recognition

STH: Spatio-Temporal Hybrid Convolution for Efficient Action Recognition

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

An Efficient Lightweight Spatio-temporal Attention Module for Action Recognition.

Efficient spatio-temporal network for action recognition

STM: SpatioTemporal and Motion Encoding for Action Recognition.

An End to End Framework with Adaptive Spatio-Temporal Attention Module for Human Action Recognition.

TEINet: Towards an Efficient Architecture for Video Recognition.

STCA: an action recognition network with spatio-temporal convolution and attention

TSI: Temporal Saliency Integration for Video Action Recognition

Video-ception Network: Towards Multi-Scale Efficient Asymmetric Spatial-Temporal Interactions

Spatio-Temporal Adaptive Network with Bidirectional Temporal Difference for Action Recognition

Spatio-Temporal Attention Networks for Action Recognition and Detection

SAST: Learning Semantic Action-Aware Spatial-Temporal Features for Efficient Action Recognition

Grouped Spatial-Temporal Aggregation for Efficient Action Recognition

Channel Attention Module for Efficient Action Recognition