Abstract:Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

Multimodel emotion analysis in response to multimedia

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Multimodal emotion recognition using EEG and eye tracking data.

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Multimodal Emotion Recognition by Combining Physiological Signals and Facial Expressions: a Preliminary Study.

EmotionMeter: A Multimodal Framework for Recognizing Human Emotions.

Modeling And Application Of Multimodal Affective User Interface With Multimedia

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Classification of Five Emotions from EEG and Eye Movement Signals: Complementary Representation Properties

Multimodal Emotion Recognition from Eye Image, Eye Movement and EEG Using Deep Neural Networks

Combining Eye Movements and EEG to Enhance Emotion Recognition.

Multi-Target Positive Emotion Recognition from EEG Signals

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

EffMulti: Efficiently Modeling Complex Multimodal Interactions for Emotion Analysis

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

MMPosE: Movie-induced Multi-label Positive Emotion Classification Through EEG Signals

Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Multidimensional Feature in Emotion Recognition Based on Multi-Channel EEG Signals.

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition