Abstract:Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

A Lightweight Multi-modal Emotion Recognition Network Based on Multi-task Learning

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Hybrid Multi-Task Learning for End-To-End Multimodal Emotion Recognition

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

A Three-stage multimodal emotion recognition network based on text low-rank fusion

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Multi-modal fusion network with complementarity and importance for emotion recognition

Multimodal Emotional Classification Based on Meaningful Learning

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Constructing multi-modal emotion recognition model based on convolutional neural network

Simplifying Multimodal Emotion Recognition with Single Eye Movement Modality

Multi-head attention fusion networks for multi-modal speech emotion recognition

Multimodal emotion recognition from facial expression and speech based on feature fusion

Facial Emotion Recognition Combining Auxiliary Classifiers and Multiscale CBAM Attention Mechanisms

Multi-Modal Emotion Recognition Based on Wavelet Transform and BERT-RoBERTa: An Innovative Approach Combining Enhanced BiLSTM and Focus Loss Function

Multi-modal Attention for Speech Emotion Recognition

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Multi-Attention Module for Dynamic Facial Emotion Recognition

A multimodal shared network with a cross-modal distribution constraint for continuous emotion recognition