Abstract:Video analysis is an important branch of computer vision due to its wide applications, ranging from video surveillance, video indexing, and retrieval to human computer interaction. All of the applications are based on a good video representation, which encodes video content into a feature vector with fixed length. Most existing methods treat video as a flat image sequence, but from our observations we argue that video is an information-intensive media with intrinsic hierarchical structure, which is largely ignored by previous approaches. Therefore, in this work, we represent the hierarchical structure of video with multiple granularities including, from short to long, single frame, consecutive frames (motion), short clip, and the entire video. Furthermore, we propose a novel deep learning framework to model each granularity individually. Specifically, we model the frame and motion granularities with 2D convolutional neural networks and model the clip and video granularities with 3D convolutional neural networks. Long Short-Term Memory networks are applied on the frame, motion, and clip to further exploit the long-term temporal clues. Consequently, the whole framework utilizes multi-stream CNNs to learn a hierarchical representation that captures spatial and temporal information of video. To validate its effectiveness in video analysis, we apply this video representation to action recognition task. We adopt a distribution-based fusion strategy to combine the decision scores from all the granularities, which are obtained by using a softmax layer on the top of each stream. We conduct extensive experiments on three action benchmarks (UCF101, HMDB51, and CCV) and achieve competitive performance against several state-of-the-art methods.

Deep Local Video Feature for Action Recognition

Local Feature Analysis for real-time Action Recognition.

Joint Multi-Scale Residual and Motion Feature Learning for Action Recognition.

Deeply-Supervised CNN Model for Action Recognition with Trainable Feature Aggregation

Unsupervised Deep Learning of Mid-Level Video Representation for Action Recognition.

Action-Stage Emphasized Spatiotemporal VLAD for Video Action Recognition

Action Recognition By Learning Deep Multi-Granular Spatio-Temporal Video Representation

End-to-end Video-level Representation Learning for Action Recognition

Learning Hierarchical Video Representation for Action Recognition

Action Recognition Using Co-trained Deep Convolutional Neural Networks.

Action Recognition Using Multiple Pooling Strategies of CNN Features

Action Recognition in Videos with Temporal Segments Fusions

Exploiting Local Feature Fusion for Action Recognition.

Local-aware spatio-temporal attention network with multi-stage feature fusion for human action recognition

3D Convolutional Two-Stream Network for Action Recognition in Videos

Recognizing micro actions in videos by learning multi-layer local features

Sparse Feature Auto-Combination Deep Network for Video Action Recognition

Action Recognition by Learning Locally Adaptive Classifiers

Action Recognition with Joint Attention on Multi-Level Deep Features

Stratified Pooling Based Deep Convolutional Neural Networks for Human Action Recognition

Integrating Temporal and Spatial Attention for Video Action Recognition