Abstract:Video analysis is an important branch of computer vision due to its wide applications, ranging from video surveillance, video indexing, and retrieval to human computer interaction. All of the applications are based on a good video representation, which encodes video content into a feature vector with fixed length. Most existing methods treat video as a flat image sequence, but from our observations we argue that video is an information-intensive media with intrinsic hierarchical structure, which is largely ignored by previous approaches. Therefore, in this work, we represent the hierarchical structure of video with multiple granularities including, from short to long, single frame, consecutive frames (motion), short clip, and the entire video. Furthermore, we propose a novel deep learning framework to model each granularity individually. Specifically, we model the frame and motion granularities with 2D convolutional neural networks and model the clip and video granularities with 3D convolutional neural networks. Long Short-Term Memory networks are applied on the frame, motion, and clip to further exploit the long-term temporal clues. Consequently, the whole framework utilizes multi-stream CNNs to learn a hierarchical representation that captures spatial and temporal information of video. To validate its effectiveness in video analysis, we apply this video representation to action recognition task. We adopt a distribution-based fusion strategy to combine the decision scores from all the granularities, which are obtained by using a softmax layer on the top of each stream. We conduct extensive experiments on three action benchmarks (UCF101, HMDB51, and CCV) and achieve competitive performance against several state-of-the-art methods.

Action Recognition by Hidden Temporal Models

Action Recognition By Learning Deep Multi-Granular Spatio-Temporal Video Representation

Action recognition with temporal scale-invariant deep learning framework

Online Robust Action Recognition Based on a Hierarchical Model

Temporal Distinct Representation Learning for Action Recognition

Temporal Segment Networks for Action Recognition in Videos

Human Action Recognition Using Deep Learning Methods.

A Temporal Order Modeling Approach to Human Action Recognition from Multimodal Sensor Data.

Action Recognition Based on Latent Task Learning

Action Recognition by Hierarchical Mid-level Action Elements

Temporal-Spatial Mapping for Action Recognition

Combining Sparse And Dense Descriptors With Temporal Semantic Structures For Robust Human Action Recognition

Action recognition using a hierarchy of feature groups

Human Action Recognition Based on Selected Spatio-Temporal Features Via Bidirectional LSTM

Human Activity Recognition based on Dynamic Spatio-Temporal Relations

Video Based Action Recognition Using Spatial and Temporal Feature

Learning Spatio-Temporal Features for Action Recognition from the Side of the Video

A Hierarchical Spatio-Temporal Model for Human Activity Recognition.

Action recognition with hierarchical convolutional neural networks features and bi-directional long short-term memory model

Learning Hierarchical Video Representation for Action Recognition

Action Recognition by Latent Duration Model