Abstract:Videos are inherently multimodal. This paper studies the problem of how to fully exploit the abundant multimodal clues for improved video categorization. We introduce a hybrid deep learning framework that integrates useful clues from multiple modalities, including static spatial appearance information, motion patterns within a short time window, audio information as well as long-range temporal dynamics. More specifically, we utilize three Convolutional Neural Networks (CNNs) operating on appearance, motion and audio signals to extract their corresponding features. We then employ a feature fusion network to derive a unified representation with an aim to capture the relationships among features. Furthermore, to exploit the long-range temporal dynamics in videos, we apply two Long Short Term Memory networks with extracted appearance and motion features as inputs. Finally, we also propose to refine the prediction scores by leveraging contextual relationships among video semantics. The hybrid deep learning framework is able to exploit a comprehensive set of multimodal features for video classification. Through an extensive set of experiments, we demonstrate that (1) LSTM networks which model sequences in an explicitly recurrent manner are highly complementary with CNN models; (2) the feature fusion network which produces a fused representation through modeling feature relationships outperforms alternative fusion strategies; (3) the semantic context of video classes can help further refine the predictions for improved performance. Experimental results on two challenging benchmarks, the UCF-101 and the Columbia Consumer Videos (CCV), provide strong quantitative evidence that our framework achieves promising results: $93.1\%$ on the UCF-101 and $84.5\%$ on the CCV, outperforming competing methods with clear margins.

Learning Frame Relevance for Video Classification

Temporal Complementarity-Guided Reinforcement Learning for Image-to-Video Person Re-Identification

Are Binary Annotations Sufficient? Video Moment Retrieval Via Hierarchical Uncertainty-Based Active Learning

Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification

Local-Global Associative Frame Assemble in Video Re-ID

Multimodal Deep Representation Learning for Video Classification

Active Learning for Video Classification with Frame Level Queries

Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

New Fuzzy-Mass Based Features for Video Image Type Categorization

A Short Video Classification Framework Based on Cross-Modal Fusion

Fine-Grained Video Categorization with Redundancy Reduction Attention.

A New Multi-Instance Learning Scheme For Scene Categorization Using Information Bottleneck Theory

VIDEO CLASSIFICATION WITH MULTIPLE FEATURE DISTANCE LEARNING

An Online-Optimized Incremental Learning Framework for Video Semantic Classification

Towards Good Practices for Multi-modal Fusion in Large-scale Video Classification

Discriminative feature extraction for video person re-identification via multi-task network

Context-aware focal alignment network for micro-video multi-label classification

Deep Unsupervised Key Frame Extraction for Efficient Video Classification

Frame Aggregation and Multi-Modal Fusion Framework for Video-Based Person Recognition

Boosting Video Representation Learning with Multi-Faceted Integration

Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification