Abstract:Humans express their emotions in a variety of ways, which inspires research on multimodal fusion-based emotion recognition that utilizes different modalities to achieve information complementation. However, extracting deep emotional features from different modalities and fusing them remain a challenging task. It is essential to exploit the advantages of different extraction and fusion approaches to capture the emotional information contained within and across modalities. In this paper, we present a novel multimodal emotion recognition framework called multimodal emotion recognition based on cascaded multichannel and hierarchical fusion (CMC-HF), where visual, speech, and text signals are simultaneously utilized as multimodal inputs. First, three cascaded channels based on deep learning technology perform feature extraction for the three modalities separately to enhance deeper information extraction ability within each modality and improve recognition performance. Second, an improved hierarchical fusion module is introduced to promote intermodality interactions of three modalities and further improve recognition and classification accuracy. Finally, to validate the effectiveness of the designed CMC-HF model, some experiments are conducted to evaluate two benchmark datasets, IEMOCAP and CMU-MOSI. The results show that we achieved an almost 2%∼3.2% increase in accuracy of the four classes for the IEMOCAP dataset as well as an improvement of 0.9%∼2.5% in the average class accuracy for the CMU-MOSI dataset when compared to the existing state-of-the-art methods. The ablation experimental results indicate that the cascaded feature extraction method and the hierarchical fusion method make a significant contribution to multimodal emotion recognition, suggesting that the three modalities contain deeper information interactions of both intermodality and intramodality. Hence, the proposed model has better overall performance and achieves higher recognition efficiency and better robustness.

Multimodal Emotion Recognition Based on Feature Selection and Extreme Learning Machine in Video Clips.

Emotion Recognition in Videos via Fusing Multimodal Features.

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Multimodal Fusion Using Kernel-Based Elm For Video Emotion Recognition

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Multi-Modal Emotion Recognition Fusing Video and Audio

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Multimodal emotion recognition from facial expression and speech based on feature fusion

Combining Multimodal Features with Hierarchical Classifier Fusion for Emotion Recognition in the Wild.

Multimodal emotion recognition using EEG and eye tracking data.

Combining Multimodal Features Within A Fusion Network For Emotion Recognition In The Wild

Multimodal Emotion Recognition by Extracting Common and Modality-Specific Information.

Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Multimodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning

Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

Research on Multimodal Emotion Recognition Based on Fusion of Electroencephalogram and Electrooculography

Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers

Multimodal interaction enhanced representation learning for video emotion recognition

Context-aware Multimodal Fusion for Emotion Recognition

An autoencoder-based feature level fusion for speech emotion recognition

Context-Aware Based Visual-Audio Feature Fusion for Emotion Recognition