Abstract:Humans express their emotions in a variety of ways, which inspires research on multimodal fusion-based emotion recognition that utilizes different modalities to achieve information complementation. However, extracting deep emotional features from different modalities and fusing them remain a challenging task. It is essential to exploit the advantages of different extraction and fusion approaches to capture the emotional information contained within and across modalities. In this paper, we present a novel multimodal emotion recognition framework called multimodal emotion recognition based on cascaded multichannel and hierarchical fusion (CMC-HF), where visual, speech, and text signals are simultaneously utilized as multimodal inputs. First, three cascaded channels based on deep learning technology perform feature extraction for the three modalities separately to enhance deeper information extraction ability within each modality and improve recognition performance. Second, an improved hierarchical fusion module is introduced to promote intermodality interactions of three modalities and further improve recognition and classification accuracy. Finally, to validate the effectiveness of the designed CMC-HF model, some experiments are conducted to evaluate two benchmark datasets, IEMOCAP and CMU-MOSI. The results show that we achieved an almost 2%∼3.2% increase in accuracy of the four classes for the IEMOCAP dataset as well as an improvement of 0.9%∼2.5% in the average class accuracy for the CMU-MOSI dataset when compared to the existing state-of-the-art methods. The ablation experimental results indicate that the cascaded feature extraction method and the hierarchical fusion method make a significant contribution to multimodal emotion recognition, suggesting that the three modalities contain deeper information interactions of both intermodality and intramodality. Hence, the proposed model has better overall performance and achieves higher recognition efficiency and better robustness.

A twin disentanglement Transformer Network with Hierarchical-Level Feature Reconstruction for robust multimodal emotion recognition

Modality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

TDFNet: Transformer-Based Deep-Scale Fusion Network for Multimodal Emotion Recognition

Token-disentangling Mutual Transformer for multimodal emotion recognition

A multimodal shared network with a cross-modal distribution constraint for continuous emotion recognition

Multilevel Transformer For Multimodal Emotion Recognition

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Fine-grained Disentangled Representation Learning for Multimodal Emotion Recognition

Residual multimodal Transformer for expression‐EEG fusion continuous emotion recognition

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Enhancing Emotion Recognition in Incomplete Data: A Novel Cross-Modal Alignment, Reconstruction, and Refinement Framework

A multi-stage dynamical fusion network for multimodal emotion recognition

Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition

Towards Learning a Joint Representation from Transformer in Multimodal Emotion Recognition

CMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation

A Multi-Level Circulant Cross-Modal Transformer for Multimodal Speech Emotion Recognition

MMTrans-MT: A Framework for Multimodal Emotion Recognition Using Multitask Learning

TMFER: Multimodal Fusion Emotion Recognition Algorithm Based on Transformer

A Deep Spatiotemporal Interaction Network for Multimodal Sentimental Analysis and Emotion Recognition

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations