Abstract:Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

Mandarin Emotion Recognition Based on Multifractal Theory Towards Human-Robot Interaction

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Emotional Speech Clustering Based Robust Speaker Recognition System

Multi-Modal Based Fuzzy Atmosfield in Human-Robot Interaction

Emotional States Based 3-D Fuzzy Atmosfield for Casual Communication Between Humans and Robots

Two-layer fuzzy multiple random forest for speech emotion recognition in human-robot interaction

Dynamic Emotion Understanding in Human–Robot Interaction Based on Two-Layer Fuzzy SVR-TS Model

A Facial Expression Emotion Recognition Based Human-robot Interaction System

Speech Emotion Recognition Based on Linear Discriminant Analysis and Support Vector Machine Decision Tree

Fractal Dimension Pattern Based Multiresolution Analysis for Rough Estimator of Person-Dependent Audio Emotion Recognition

EmoEars: an emotion recognition system for mandarin speech

Speech Emotion Recognition Based on Feature Selection and Extreme Learning Machine Decision Tree

A Multimodal Emotional Communication Based Humans-Robots Interaction System

Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Multi-Modal Emotion Recognition Based on Wavelet Transform and BERT-RoBERTa: An Innovative Approach Combining Enhanced BiLSTM and Focus Loss Function

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Emotional Communication Robot Based on 3D Face Model and ASR Technology

Application of Speech Emotion Recognition in Intelligent Household Robot

Mismatched Feature Detection with Finer Granularity for Emotional Speaker Recognition.