Abstract:Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

MRAC'23: 1st International Workshop on Multimodal and Responsible Affective Computing

MRAC Track 1: 2nd Workshop on Multimodal, Generative and Responsible Affective Computing

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

A Multimodal Emotional Communication Based Humans-Robots Interaction System

FME '23: 3rd Facial Micro-Expression Workshop

A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective

UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition

Multimodal Emotion Recognition by Extracting Common and Modality-Specific Information.

Modeling And Application Of Multimodal Affective User Interface With Multimedia

MEC 2016: The Multimodal Emotion Recognition Challenge of CCPR 2016.

Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning.

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Modality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition

ASMMC-MMAC 2018: The Joint Workshop of 4th the Workshop on Affective Social Multimedia Computing and first Multi-Modal Affective Computing of Large-Scale Multimedia Data Workshop.

MultiEMO: an Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations.