Abstract:Most existing action recognition approaches directly leverage the video-level features to recognize human actions from videos. Although these methods have made remarkable progress, the accuracy is still unsatisfied. When the test video involves complex backgrounds and activities, existing methods usually suffer from a significant drop in accuracy. Human action is inherently a high-level concept. Merely applying a video classification model without a detailed semantic understanding of the video content, e.g., objects, scene context, object motions, object interactions, is inadequate to tackle the challenges for action recognition. Fine-level semantic understanding of videos generates elementary semantic concepts from the raw video data, such as the semantics of objects and background regions. It can be employed to bridge the gap between the raw video data and the high-level concept of human actions. In this work, we leverage dense semantic segmentation masks, which encode rich semantic details, provide extra information for the network training, and improve the performance of action recognition. We propose a novel deep architecture which is named as Dense Semantics-Assisted Convolutional Neural Networks (DSA-CNNs) to effectively utilize dense semantic information of video by a bottom-up attention way in the spatial stream, while by the way of branch fusion in the temporal stream. To verify the effectiveness of our approach, we conduct extensive experiments on publicly available datasets – UCF101, HMDB51, and Kinetics. The experimental results demonstrate that our approach substantially improves existing methods and achieves very competitive performance. It also shows that our approach is superior to other related methods that utilize extra information for action recognition.

Latent Semantic Learning by Efficient Sparse Coding with Hypergraph Regularization.

Latent Semantic Learning with Structured Sparse Representation for Human Action Recognition

Spectral Learning of Latent Semantics for Action Recognition

Encoding Learning Network Combined with Feature Similarity Constraints for Human Action Recognition

Constructing Visual Vocabularies Using Sparse Coding for Action Recognition

Learning Latent Spatio-Temporal Compositional Model for Human Action Recognition

A Compact Representation of Human Actions by Sliding Coordinate Coding

Dense Semantics-Assisted Networks For Video Action Recognition

Combining Sparse And Dense Descriptors With Temporal Semantic Structures For Robust Human Action Recognition

Efficient Probabilistic Latent Semantic Analysis with Sparsity Control

Human Action Recognition Using Sparse Representation

Sparse Semi-Supervised Action Recognition with Active Learning

Semi-Supervised Multiple Feature Analysis for Action Recognition

Spatio-temporal Laplacian Pyramid Coding for Action Recognition.

P-Laplacian Regularized Sparse Coding for Human Activity Recognition

Action Recognition by Exploring Data Distribution and Feature Correlation

Linguistic-Driven Partial Semantic Relevance Learning for Skeleton-Based Action Recognition

Collaborative Spatio-temporal Feature Learning for Video Action Recognition

Video Action Recognition with Attentive Semantic Units

Deep set conditioned latent representations for action recognition

Action Recognition By Learning Deep Multi-Granular Spatio-Temporal Video Representation