Abstract:Emotion recognition plays an important role in intelligent human-computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram (EEG) and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a decision level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network (CNN) based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine (SVM) and a subject-independent model based on long short-term memory network (LSTM). In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals (DEAP) dataset and the Multimodal Database for Affect Recognition and Implicit Tagging (MAHNOB-HCI) dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated.

SEED-VII: A Multimodal Dataset of Six Basic Emotions with Continuous Labels for Emotion Recognition

Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

Emotion Recognition With Audio, Video, EEG, and EMG: A Dataset and Baseline Approaches

A Multimodal Dataset for Mixed Emotion Recognition

Exploring Emotions in EEG: Deep Learning Approach with Feature Fusion

HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild

HEU Emotion: a Large-Scale Database for Multimodal Emotion Recognition in the Wild

Emotions: A Large-Scale Dataset for Emotion Recognition in Short Videos

Multimodal Emotion Recognition Based on EEG and EOG Signals Evoked by the Video-Odor Stimuli

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition

MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition

EEG-SCMM: Soft Contrastive Masked Modeling for Cross-Corpus EEG-Based Emotion Recognition

Research on Multimodal Emotion Recognition Based on Fusion of Electroencephalogram and Electrooculography

A Publicly Available Multimodal Emotional Dataset Triggered by Both Videos and Odors

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

EmotionMeter: A Multimodal Framework for Recognizing Human Emotions.

Bridging Discrete and Continuous: A Multimodal Strategy for Complex Emotion Detection

A Model for EEG-Based Emotion Recognition: CNN-Bi-LSTM with Attention Mechanism