Abstract:Videos are inherently multimodal. This paper studies the problem of how to fully exploit the abundant multimodal clues for improved video categorization. We introduce a hybrid deep learning framework that integrates useful clues from multiple modalities, including static spatial appearance information, motion patterns within a short time window, audio information as well as long-range temporal dynamics. More specifically, we utilize three Convolutional Neural Networks (CNNs) operating on appearance, motion and audio signals to extract their corresponding features. We then employ a feature fusion network to derive a unified representation with an aim to capture the relationships among features. Furthermore, to exploit the long-range temporal dynamics in videos, we apply two Long Short Term Memory networks with extracted appearance and motion features as inputs. Finally, we also propose to refine the prediction scores by leveraging contextual relationships among video semantics. The hybrid deep learning framework is able to exploit a comprehensive set of multimodal features for video classification. Through an extensive set of experiments, we demonstrate that (1) LSTM networks which model sequences in an explicitly recurrent manner are highly complementary with CNN models; (2) the feature fusion network which produces a fused representation through modeling feature relationships outperforms alternative fusion strategies; (3) the semantic context of video classes can help further refine the predictions for improved performance. Experimental results on two challenging benchmarks, the UCF-101 and the Columbia Consumer Videos (CCV), provide strong quantitative evidence that our framework achieves promising results: $93.1\%$ on the UCF-101 and $84.5\%$ on the CCV, outperforming competing methods with clear margins.

Multimodal E-Commerce Product Classification Using Hierarchical Fusion

Product Images Classification with Mul-tiple Features Combination

A Multimodal Late Fusion Model for E-Commerce Product Classification

Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce

Knowledge Perceived Multi-modal Pretraining in E-commerce

Image and Encoded Text Fusion for Multi-Modal Classification

Neutrosophic-CNN-based image and text fusion for multimodal classification

Large Scale Multimodal Classification Using an Ensemble of Transformer Models and Co-Attention

Multimodal Approach for Harmonized System Code Prediction

Multimodal Sentiment Analysis Based on Composite Hierarchical Fusion

Multimodal data fusion using signal/image processing methods for multi-class machine learning

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

End-to-end multi-modal product matching in fashion e-commerce

Deep Multimodal Data Fusion

Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification

Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

A unified multimodal classification framework based on deep metric learning

Multimodal Classification for Analysing Social Media

Multimodal Metadata Assignment for Cultural Heritage Artifacts

CMSE: Cross-Modal Semantic Enhancement Network for Classification of Hyperspectral and LiDAR Data

A review of deep learning-based information fusion techniques for multimodal medical image classification