Abstract:Humans express their emotions in a variety of ways, which inspires research on multimodal fusion-based emotion recognition that utilizes different modalities to achieve information complementation. However, extracting deep emotional features from different modalities and fusing them remain a challenging task. It is essential to exploit the advantages of different extraction and fusion approaches to capture the emotional information contained within and across modalities. In this paper, we present a novel multimodal emotion recognition framework called multimodal emotion recognition based on cascaded multichannel and hierarchical fusion (CMC-HF), where visual, speech, and text signals are simultaneously utilized as multimodal inputs. First, three cascaded channels based on deep learning technology perform feature extraction for the three modalities separately to enhance deeper information extraction ability within each modality and improve recognition performance. Second, an improved hierarchical fusion module is introduced to promote intermodality interactions of three modalities and further improve recognition and classification accuracy. Finally, to validate the effectiveness of the designed CMC-HF model, some experiments are conducted to evaluate two benchmark datasets, IEMOCAP and CMU-MOSI. The results show that we achieved an almost 2%∼3.2% increase in accuracy of the four classes for the IEMOCAP dataset as well as an improvement of 0.9%∼2.5% in the average class accuracy for the CMU-MOSI dataset when compared to the existing state-of-the-art methods. The ablation experimental results indicate that the cascaded feature extraction method and the hierarchical fusion method make a significant contribution to multimodal emotion recognition, suggesting that the three modalities contain deeper information interactions of both intermodality and intramodality. Hence, the proposed model has better overall performance and achieves higher recognition efficiency and better robustness.

User Emotion Recognition Based on Multidimensional Data Feature Fusion

Emotion Recognition in Videos via Fusing Multimodal Features.

Multimodal Emotion Recognition Based on Feature Fusion.

A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition

Enhancing emotion recognition using multimodal fusion of physiological, environmental, personal data

Emotion Recognition Model Based on Multimodal Decision Fusion

MoodExplorer

An Improved Multimodal Dimension Emotion Recognition Based on Different Fusion Methods

Behavioral and Physiological Signals-Based Deep Multimodal Approach for Mobile Emotion Recognition

Arbuscular Mycorrhizae Effects on Growth of Two Hawaiian Species: Indigenous Osteomeles anthyllidifolia (Rosaceae) and Invasive Psidium cattleianum (Myrtaceae)

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

Feature-level fusion of multimodal physiological signals for emotion recognition

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Wearable Biosensor Network Enabled Multimodal Daily-Life Emotion Recognition Employing Reputation-Driven Imbalanced Fuzzy Classification

End-to-End Multimodal Emotion Recognition Based on Facial Expressions and Remote Photoplethysmography Signals

Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

Research on Emotional Interaction and User Cognition

Multimodal Emotion Recognition Based on Facial Expressions, Speech, and Body Gestures

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Emotion Recognition Based on Weighted Fusion Strategy of Multichannel Physiological Signals