Abstract:Emotion recognition plays an important role in intelligent human-computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram (EEG) and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a decision level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network (CNN) based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine (SVM) and a subject-independent model based on long short-term memory network (LSTM). In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals (DEAP) dataset and the Multimodal Database for Affect Recognition and Implicit Tagging (MAHNOB-HCI) dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated.

MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

An EEG-Based Multi-Modal Emotion Database with Both Posed and Authentic Facial Actions for Emotion Analysis

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Multi-modal emotion recognition using EEG and speech signals

HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

HEU Emotion: a Large-Scale Database for Multimodal Emotion Recognition in the Wild

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

DEAP: A Database for Emotion Analysis ;Using Physiological Signals

Emotion Recognition With Audio, Video, EEG, and EMG: A Dataset and Baseline Approaches

MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition

ECSMP: A dataset on emotion, cognition, sleep, and multi-model physiological signals

A Multimodal Dataset for Mixed Emotion Recognition

MEDA-CBLSTM: Data Acquisition, Processing and Emotion Recognition Based on Multi-Layer EDA

Ensemble Emotion Recognizing with Multiple Modal Physiological Signals

SEED-VII: A Multimodal Dataset of Six Basic Emotions with Continuous Labels for Emotion Recognition

Development of Multi-modal Physiological Signals Acquisition Platform and Its Application in Emotional Recognition Based on a Small Number of Samples

Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels

MEMOS: A Multi-modal Emotion Stream Database for Temporal Spontaneous Emotional State Detection

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression