Abstract:Most existing action recognition approaches directly leverage the video-level features to recognize human actions from videos. Although these methods have made remarkable progress, the accuracy is still unsatisfied. When the test video involves complex backgrounds and activities, existing methods usually suffer from a significant drop in accuracy. Human action is inherently a high-level concept. Merely applying a video classification model without a detailed semantic understanding of the video content, e.g., objects, scene context, object motions, object interactions, is inadequate to tackle the challenges for action recognition. Fine-level semantic understanding of videos generates elementary semantic concepts from the raw video data, such as the semantics of objects and background regions. It can be employed to bridge the gap between the raw video data and the high-level concept of human actions. In this work, we leverage dense semantic segmentation masks, which encode rich semantic details, provide extra information for the network training, and improve the performance of action recognition. We propose a novel deep architecture which is named as Dense Semantics-Assisted Convolutional Neural Networks (DSA-CNNs) to effectively utilize dense semantic information of video by a bottom-up attention way in the spatial stream, while by the way of branch fusion in the temporal stream. To verify the effectiveness of our approach, we conduct extensive experiments on publicly available datasets – UCF101, HMDB51, and Kinetics. The experimental results demonstrate that our approach substantially improves existing methods and achieves very competitive performance. It also shows that our approach is superior to other related methods that utilize extra information for action recognition.

DB-LSTM: Densely-connected Bi-directional LSTM for Human Action Recognition

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Human Action Recognition From Digital Videos Based on Deep Learning.

Human Action Recognition Using Deep Learning Methods.

Lattice Long Short-Term Memory for Human Action Recognition

Deep spatiotemporal LSTM network with temporal pattern feature for 3D human action recognition

Action Recognition By Learning Deep Multi-Granular Spatio-Temporal Video Representation

Human Action Recognition Based on Hierarchical Multi-Scale Adaptive Conv-Long Short-Term Memory Network

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Behavior recognition based on the improved density clustering and context-guided Bi-LSTM model

Two Stream LSTM: A Deep Fusion Framework for Human Action Recognition

Human action recognition using attention based LSTM network with dilated CNN features

Deep historical long short-term memory network for action recognition

A very deep sequences learning approach for human action recognition

Dense Semantics-Assisted Networks For Video Action Recognition

Deep Learning-Based Human Action Recognition in Videos

Relational Long Short-Term Memory for Video Action Recognition

TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition

Action Recognition Algorithm of Spatio-Temporal Differential LSTM Based on Feature Enhancement

Skeleton-based Action Recognition Using LSTM and CNN

Spatio-Temporal Attention-Based LSTM Networks for 3D Action Recognition and Detection