Abstract:Humans express their emotions in a variety of ways, which inspires research on multimodal fusion-based emotion recognition that utilizes different modalities to achieve information complementation. However, extracting deep emotional features from different modalities and fusing them remain a challenging task. It is essential to exploit the advantages of different extraction and fusion approaches to capture the emotional information contained within and across modalities. In this paper, we present a novel multimodal emotion recognition framework called multimodal emotion recognition based on cascaded multichannel and hierarchical fusion (CMC-HF), where visual, speech, and text signals are simultaneously utilized as multimodal inputs. First, three cascaded channels based on deep learning technology perform feature extraction for the three modalities separately to enhance deeper information extraction ability within each modality and improve recognition performance. Second, an improved hierarchical fusion module is introduced to promote intermodality interactions of three modalities and further improve recognition and classification accuracy. Finally, to validate the effectiveness of the designed CMC-HF model, some experiments are conducted to evaluate two benchmark datasets, IEMOCAP and CMU-MOSI. The results show that we achieved an almost 2%∼3.2% increase in accuracy of the four classes for the IEMOCAP dataset as well as an improvement of 0.9%∼2.5% in the average class accuracy for the CMU-MOSI dataset when compared to the existing state-of-the-art methods. The ablation experimental results indicate that the cascaded feature extraction method and the hierarchical fusion method make a significant contribution to multimodal emotion recognition, suggesting that the three modalities contain deeper information interactions of both intermodality and intramodality. Hence, the proposed model has better overall performance and achieves higher recognition efficiency and better robustness.

Feature relationships hypergraph for multimodal recognition

Feature Correlation Hypergraph: Exploiting High-order Potentials for Multimodal Recognition

Multimodal Feature Fusion Based Hypergraph Learning Model

Optimal Multimodal Fusion for Multimedia Data Analysis

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification

Multi-Feature Fusion Via Hierarchical Regression for Multimedia Analysis

An Efficient Approach to Informative Feature Extraction from Multimodal Data

Heterogeneous feature learning network for multimodal remote sensing image collaborative classification

Dense Multimodal Fusion for Hierarchically Joint Representation

Multi-Hypergraph Learning for Incomplete Multimodality Data

Adaptive Multimodal Robust Feature Learning Based on Dual Graph-regularization

Improving Fine-grained Image Classification with Multimodal Information

Multimodal Understanding Through Correlation Maximization and Minimization

Person Recognition with HGR Maximal Correlation on Multimodal Data.

A Discriminative Vectorial Framework for Multi-modal Feature Representation

Multimodal Feature Fusion for 3D Shape Recognition and Retrieval.

Structure-Aware Multimodal Feature Fusion for RGB-D Scene Classification and Beyond

Self-Supervised Hypergraph Learning for Enhanced Multimodal Representation

A Relation-Aware Heterogeneous Graph Transformer on Dynamic Fusion for Multimodal Classification Tasks

A unified multimodal classification framework based on deep metric learning