Abstract:Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

Correlation-Driven Multi-Modality Graph Decomposition for Cross-Subject Emotion Recognition

Dense Graph Convolutional with Joint Cross-Attention Network for Multimodal Emotion Recognition

Emotion Recognition from Physiological Signals Using Multi-Hypergraph Neural Networks.

Multi-domain fusion deep graph convolution neural network for EEG emotion recognition

Physiological Signals-based Emotion Recognition via High-order Correlation Learning

A Cross-subject and Cross-modal Model for Multimodal Emotion Recognition

Convolutional Gated Recurrent Unit-Driven Multidimensional Dynamic Graph Neural Network for Subject-Independent Emotion Recognition.

Emotion Recognition Using Cross-Modal Attention from Eeg and Facial Expression

Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion Recognition

A multi-stage dynamical fusion network for multimodal emotion recognition

A Novel and Powerful Dual-Stream Multi-Level Graph Convolution Network for Emotion Recognition

Multi-source Selective Graph Domain Adaptation Network for cross-subject EEG emotion recognition

Multimodal Emotion Recognition Model using Physiological Signals

A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion Recognition

Tensor Correlation Fusion for Multimodal Physiological Signal Emotion Recognition

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Multimodal Emotion Recognition Using Deep Generalized Canonical Correlation Analysis with an Attention Mechanism

A Two-Stage Multimodal Emotion Recognition Model Based on Graph Contrastive Learning

Multi-modal Correlated Network for Emotion Recognition in Speech

Multiplex graph aggregation and feature refinement for unsupervised incomplete multimodal emotion recognition

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG