Abstract:Humans express their emotions in a variety of ways, which inspires research on multimodal fusion-based emotion recognition that utilizes different modalities to achieve information complementation. However, extracting deep emotional features from different modalities and fusing them remain a challenging task. It is essential to exploit the advantages of different extraction and fusion approaches to capture the emotional information contained within and across modalities. In this paper, we present a novel multimodal emotion recognition framework called multimodal emotion recognition based on cascaded multichannel and hierarchical fusion (CMC-HF), where visual, speech, and text signals are simultaneously utilized as multimodal inputs. First, three cascaded channels based on deep learning technology perform feature extraction for the three modalities separately to enhance deeper information extraction ability within each modality and improve recognition performance. Second, an improved hierarchical fusion module is introduced to promote intermodality interactions of three modalities and further improve recognition and classification accuracy. Finally, to validate the effectiveness of the designed CMC-HF model, some experiments are conducted to evaluate two benchmark datasets, IEMOCAP and CMU-MOSI. The results show that we achieved an almost 2%∼3.2% increase in accuracy of the four classes for the IEMOCAP dataset as well as an improvement of 0.9%∼2.5% in the average class accuracy for the CMU-MOSI dataset when compared to the existing state-of-the-art methods. The ablation experimental results indicate that the cascaded feature extraction method and the hierarchical fusion method make a significant contribution to multimodal emotion recognition, suggesting that the three modalities contain deeper information interactions of both intermodality and intramodality. Hence, the proposed model has better overall performance and achieves higher recognition efficiency and better robustness.

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

SWRR: Feature Map Classifier Based on Sliding Window Attention and High-Response Feature Reuse for Multimodal Emotion Recognition

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversations

A multi-stage dynamical fusion network for multimodal emotion recognition

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

MRFER: Multi-Channel Robust Feature Enhanced Fusion for Multi-Modal Emotion Recognition

A Dual Attention-based Modality-Collaborative Fusion Network for Emotion Recognition

MMGCN: Multimodal Fusion Via Deep Graph Convolution Network for Emotion Recognition in Conversation

Emotion Recognition Model Based on Multimodal Decision Fusion

Dynamic Interactive Multiview Memory Network for Emotion Recognition in Conversation

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Context-aware Multimodal Fusion for Emotion Recognition

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

A multimodal fusion-based deep learning framework combined with local-global contextual TCNs for continuous emotion recognition from videos

Multimodal Fused Emotion Recognition about Expression-EEG Interaction and Collaboration Using Deep Learning

Robust Facial Reactions Generation: An Emotion-Aware Framework with Modality Compensation

TDFNet: Transformer-Based Deep-Scale Fusion Network for Multimodal Emotion Recognition

Multi-Modal Emotion Recognition Fusing Video and Audio

Multi-Scale Receptive Field Graph Model for Emotion Recognition in Conversations

Topics Guided Multimodal Fusion Network for Conversational Emotion Recognition