Abstract:Videos are inherently multimodal. This paper studies the problem of how to fully exploit the abundant multimodal clues for improved video categorization. We introduce a hybrid deep learning framework that integrates useful clues from multiple modalities, including static spatial appearance information, motion patterns within a short time window, audio information as well as long-range temporal dynamics. More specifically, we utilize three Convolutional Neural Networks (CNNs) operating on appearance, motion and audio signals to extract their corresponding features. We then employ a feature fusion network to derive a unified representation with an aim to capture the relationships among features. Furthermore, to exploit the long-range temporal dynamics in videos, we apply two Long Short Term Memory networks with extracted appearance and motion features as inputs. Finally, we also propose to refine the prediction scores by leveraging contextual relationships among video semantics. The hybrid deep learning framework is able to exploit a comprehensive set of multimodal features for video classification. Through an extensive set of experiments, we demonstrate that (1) LSTM networks which model sequences in an explicitly recurrent manner are highly complementary with CNN models; (2) the feature fusion network which produces a fused representation through modeling feature relationships outperforms alternative fusion strategies; (3) the semantic context of video classes can help further refine the predictions for improved performance. Experimental results on two challenging benchmarks, the UCF-101 and the Columbia Consumer Videos (CCV), provide strong quantitative evidence that our framework achieves promising results: $93.1\%$ on the UCF-101 and $84.5\%$ on the CCV, outperforming competing methods with clear margins.

Multimodal video classification with stacked contractive autoencoders

Stacked Convolutional Deep Encoding Network for Video-Text Retrieval.

Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification

Multimodal scene classification for encoder-assisted videos

Multimodal Deep Representation Learning for Video Classification

CrossMAE: Cross Modality Masked Autoencoders for Region-Aware Audio-Visual Pretraining

Multi-Stream Multi-Class Fusion of Deep Networks for Video Classification.

Fusing Multi-Stream Deep Networks for Video Classification

Stacked Multimodal Attention Network for Context-Aware Video Captioning

Cyclic Autoencoder for Multimodal Data Alignment Using Custom Datasets

Deep Multimodal Learning: An Effective Method for Video Classification

Effective Multi-Modal Retrieval Based on Stacked Auto-Encoders

Mutual Complementarity: Multi-Modal Enhancement Semantic Learning for Micro-Video Scene Recognition

Deep Reinforcement Learning Visual-Text Attention for Multimodal Video Classification

Multi-channel Attentive Weighting of Visual Frames for Multimodal Video Classification

Multimodal interaction enhanced representation learning for video emotion recognition

Multimodal Progressive Modulation Network for Micro-video Multi-label Classification

Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification

Multi-Task Video Captioning with a Stepwise Multimodal Encoder

Tri-Modal Dense Video Captioning Based on Fine-Grained Aligned Text and Anchor-Free Event Proposals Generator

Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detection