Abstract:Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

A New Emotion Classification Method Based on JAN-VMD

JDAT: Joint-Dimension-Aware Transformer with Strong Flexibility for EEG Emotion Recognition

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

Using electroencephalogram for emotion recognition based on filter-bank long short-term memory networks

Electroencephalogram Emotion Recognition Using Combined Features in Variational Mode Decomposition Domain

Emotion Classification Using Deep Neural Networks and Emotional Patches.

EEG-Based Emotion Recognition via Joint Domain Adaptation and Semi-supervised RVFL Network

Emotion recognition with convolutional neural network and EEG-based EFDMs

Emotion Recognition from Spatiotemporal EEG Representations with Hybrid Convolutional Recurrent Neural Networks Via Wearable Multi-Channel Headset

EEG-based cross-subject emotion recognition using multi-source domain transfer learning

Joint Distribution Adaptation Network for Multi-source Electroencephalogram-based Emotion Recognition

Multi-source joint domain adaptation for cross-subject and cross-session emotion recognition from electroencephalography

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Subject-Independent Emotion Recognition of EEG Signals Based on Dynamic Empirical Convolutional Neural Network.

Instance-representation transfer method based on joint distribution and deep adaptation for EEG emotion recognition

Eeg-Based Emotion Classification Using Innovative Features And Combined Svm And Hmm Classifier

Improving Session-to-session Transfer Performance of Emotion Recognition Using Adaptive Support Vector Machine

MS-MDA: Multisource Marginal Distribution Adaptation for Cross-Subject and Cross-Session EEG Emotion Recognition

Multi-Modal Domain Adaptation Variational Autoencoder for EEG-Based Emotion Recognition

Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks

Emotion Analysis: Bimodal Fusion of Facial Expressions and EEG