Abstract:Most existing action recognition approaches directly leverage the video-level features to recognize human actions from videos. Although these methods have made remarkable progress, the accuracy is still unsatisfied. When the test video involves complex backgrounds and activities, existing methods usually suffer from a significant drop in accuracy. Human action is inherently a high-level concept. Merely applying a video classification model without a detailed semantic understanding of the video content, e.g., objects, scene context, object motions, object interactions, is inadequate to tackle the challenges for action recognition. Fine-level semantic understanding of videos generates elementary semantic concepts from the raw video data, such as the semantics of objects and background regions. It can be employed to bridge the gap between the raw video data and the high-level concept of human actions. In this work, we leverage dense semantic segmentation masks, which encode rich semantic details, provide extra information for the network training, and improve the performance of action recognition. We propose a novel deep architecture which is named as Dense Semantics-Assisted Convolutional Neural Networks (DSA-CNNs) to effectively utilize dense semantic information of video by a bottom-up attention way in the spatial stream, while by the way of branch fusion in the temporal stream. To verify the effectiveness of our approach, we conduct extensive experiments on publicly available datasets – UCF101, HMDB51, and Kinetics. The experimental results demonstrate that our approach substantially improves existing methods and achieves very competitive performance. It also shows that our approach is superior to other related methods that utilize extra information for action recognition.

FENet: An Efficient Feature Excitation Network for Video-based Human Action Recognition

Encoding Learning Network Combined with Feature Similarity Constraints for Human Action Recognition

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Frequency Enhancement Network for Efficient Compressed Video Action Recognition

ACTION-Net: Multipath Excitation for Action Recognition

TEINet: Towards an Efficient Architecture for Video Recognition.

DC3D: A Video Action Recognition Network Based on Dense Connection

FEXNet: Foreground Extraction Network for Human Action Recognition

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Human Action Recognition Based on Three-Stream Network with Frame Sequence Features

Fine-gained Motion Enhancement for Action Recognition: Focusing on Action-Related Regions

FEASE: Feature Selection and Enhancement Networks for Action Recognition

Efficient spatio-temporal network for action recognition

Cross-Fiber Spatial-Temporal Co-enhanced Networks for Video Action Recognition

Fine-Tuned Temporal Dense Sampling with 1D Convolutional Neural Network for Human Action Recognition

Local-aware spatio-temporal attention network with multi-stage feature fusion for human action recognition

Attentional Fused Temporal Transformation Network for Video Action Recognition.

Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks

Human Action Recognition Based on Improved Fusion Attention CNN and RNN

Two-Stream 3-D Convnet Fusion for Action Recognition in Videos with Arbitrary Size and Length

Dense Semantics-Assisted Networks For Video Action Recognition