Abstract:Videos are inherently multimodal. This paper studies the problem of how to fully exploit the abundant multimodal clues for improved video categorization. We introduce a hybrid deep learning framework that integrates useful clues from multiple modalities, including static spatial appearance information, motion patterns within a short time window, audio information as well as long-range temporal dynamics. More specifically, we utilize three Convolutional Neural Networks (CNNs) operating on appearance, motion and audio signals to extract their corresponding features. We then employ a feature fusion network to derive a unified representation with an aim to capture the relationships among features. Furthermore, to exploit the long-range temporal dynamics in videos, we apply two Long Short Term Memory networks with extracted appearance and motion features as inputs. Finally, we also propose to refine the prediction scores by leveraging contextual relationships among video semantics. The hybrid deep learning framework is able to exploit a comprehensive set of multimodal features for video classification. Through an extensive set of experiments, we demonstrate that (1) LSTM networks which model sequences in an explicitly recurrent manner are highly complementary with CNN models; (2) the feature fusion network which produces a fused representation through modeling feature relationships outperforms alternative fusion strategies; (3) the semantic context of video classes can help further refine the predictions for improved performance. Experimental results on two challenging benchmarks, the UCF-101 and the Columbia Consumer Videos (CCV), provide strong quantitative evidence that our framework achieves promising results: $93.1\%$ on the UCF-101 and $84.5\%$ on the CCV, outperforming competing methods with clear margins.

A Hybrid Approach To News Video Classification With Multi-Modal Features

News Video Classification Using Multimodal Classifiers and Text-Biased Combination Strategies

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Look and Listen: A Multi-modality Late Fusion Approach to Scene Classification for Autonomous Machines

Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification

Integration of Multimodal Features for Video Scene Classification Based on HMM

Hybrid Svm/Hmm Approach For Audio Classification

Hybrid Support Vector Machine and General Model Approach for Audio Classification

Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

Hybrid Independent Component Analysis And Support Vector Machine Approach For Audio Classification

Hybrid grass bee optimization‐multikernal extreme learning classifier: Multimodular fusion strategy and optimal feature selection for multimodal sentiment analysis in social media videos

Multi-Stream Multi-Class Fusion of Deep Networks for Video Classification.

Multimodal Classification for Analysing Social Media

A Method For Content-Based News Story Classification In Data Mining

Multimodal sentiment analysis leveraging the strength of deep neural networks enhanced by the XGBoost classifier

Multi-modal Semantic Inconsistency Detection in Social Media News Posts

Fusing Multi-Stream Deep Networks for Video Classification

Multimodal Deep Representation Learning for Video Classification

Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention

Contrastive Graph Multimodal Model for Text Classification in Videos

Joint Audio-Visual Bi-Modal Codewords for Video Event Detection.