Abstract:Videos are inherently multimodal. This paper studies the problem of how to fully exploit the abundant multimodal clues for improved video categorization. We introduce a hybrid deep learning framework that integrates useful clues from multiple modalities, including static spatial appearance information, motion patterns within a short time window, audio information as well as long-range temporal dynamics. More specifically, we utilize three Convolutional Neural Networks (CNNs) operating on appearance, motion and audio signals to extract their corresponding features. We then employ a feature fusion network to derive a unified representation with an aim to capture the relationships among features. Furthermore, to exploit the long-range temporal dynamics in videos, we apply two Long Short Term Memory networks with extracted appearance and motion features as inputs. Finally, we also propose to refine the prediction scores by leveraging contextual relationships among video semantics. The hybrid deep learning framework is able to exploit a comprehensive set of multimodal features for video classification. Through an extensive set of experiments, we demonstrate that (1) LSTM networks which model sequences in an explicitly recurrent manner are highly complementary with CNN models; (2) the feature fusion network which produces a fused representation through modeling feature relationships outperforms alternative fusion strategies; (3) the semantic context of video classes can help further refine the predictions for improved performance. Experimental results on two challenging benchmarks, the UCF-101 and the Columbia Consumer Videos (CCV), provide strong quantitative evidence that our framework achieves promising results: $93.1\%$ on the UCF-101 and $84.5\%$ on the CCV, outperforming competing methods with clear margins.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to make full use of the rich multimodal cues in videos to improve the accuracy of video classification. Specifically, the authors focus on how to integrate static spatial appearance information, motion patterns within a short - time window, audio information, and long - time - span temporal dynamics to improve the effect of video classification. In addition, they also consider the contextual relationships between video semantics, which can further optimize the prediction results. To solve the above problems, the authors propose a hybrid deep - learning framework, which consists of the following key components: 1. **Multimodal Feature Extraction**: - Use three convolutional neural networks (CNNs) to process video frames, stacked optical - flow images, and audio signals respectively to extract corresponding features. 2. **Long - Short - Term Memory Networks (LSTMs)**: - In order to capture the long - time - span temporal dynamics in videos, the authors introduce two LSTM models, taking the extracted spatial features and motion features as inputs respectively. 3. **Feature Fusion Network**: - Through a neural network with carefully designed regularization, fuse the features of multiple modalities to obtain a unified representation form and explore the relationships between different features. 4. **Contextual Relationship Refinement**: - Use the confusion matrix to model the semantic contextual relationships between video categories, thereby further optimizing the final prediction scores. Through a series of experiments, the authors prove that this framework has achieved results significantly better than other methods on two benchmark datasets, UCF - 101 and Columbia Consumer Videos (CCV), reaching accuracies of 93.1% and 84.5% respectively. This shows the high complementarity between the LSTM and CNN models, as well as the effectiveness of the feature - fusion network and the importance of semantic context for performance improvement.

Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification

Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

Multimodal Deep Representation Learning for Video Classification

Fusing Multi-Stream Deep Networks for Video Classification

Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification

Multi-Stream Multi-Class Fusion of Deep Networks for Video Classification.

HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognition

A Short Video Classification Framework Based on Cross-Modal Fusion

Deep Multimodal Learning: An Effective Method for Video Classification

Multimodal video classification with stacked contractive autoencoders

Integration of Multimodal Features for Video Scene Classification Based on HMM

Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks.

Multimodal Deep Hierarchical Semantic-Aligned Matrix Factorization Method for Micro-Video Multi-Label Classification

Hierarchical Attention-Based Multimodal Fusion for Video Captioning

Multimodal interaction enhanced representation learning for video emotion recognition

Boosting Video Representation Learning with Multi-Faceted Integration

Structure-Aware Multimodal Feature Fusion for RGB-D Scene Classification and Beyond

Neural Multimodal Cooperative Learning Toward Micro-Video Understanding

Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval

Hybrid Improvements in Multimodal Analysis for Deep Video Understanding

A unified multimodal classification framework based on deep metric learning