Abstract:Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

Multimodal Emotion Recognition in Noisy Environment Based on Progressive Label Revision

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Facial Emotion Recognition with Noisy Multi-task Annotations

MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Multimodal Emotional Classification Based on Meaningful Learning

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Learning Noise-Robust Joint Representation for Multimodal Emotion Recognition under Incomplete Data Scenarios

MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning.

Explainable Multimodal Emotion Reasoning: a Promising Way to Open-set Emotion Recognition

SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition

Multi-channel EEG-based Emotion Recognition in the Presence of Noisy Labels

First-order Multi-label Learning with Cross-modal Interactions for Multimodal Emotion Recognition

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Learning Robust Multi-Modal Representation for Multi-Label Emotion Recognition Via Adversarial Masking and Perturbation

Multimodal Emotion Recognition and Sentiment Analysis via Attention Enhanced Recurrent Model

Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism

Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropout

Improving Multimodal Emotion Recognition by Leveraging Acoustic Adaptation and Visual Alignment

A multimodal shared network with a cross-modal distribution constraint for continuous emotion recognition