Abstract:Emotion recognition plays an important role in intelligent human-computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram (EEG) and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a decision level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network (CNN) based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine (SVM) and a subject-independent model based on long short-term memory network (LSTM). In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals (DEAP) dataset and the Multimodal Database for Affect Recognition and Implicit Tagging (MAHNOB-HCI) dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated.

EMTk -- The Emotion Mining Toolkit

EmoTxt: A Toolkit for Emotion Recognition from Text

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Emolysis: A Multimodal Open-Source Group Emotion Analysis and Visualization Toolkit

An Emotion based Sentiment Analysis on Twitter Dataset

EmT: A Novel Transformer for Generalized Cross-subject EEG Emotion Recognition

EmTract: Extracting Emotions from Social Media

General Purpose Textual Sentiment Analysis and Emotion Detection Tools

A Benchmark Study on Sentiment Analysis for Software Engineering Research

Towards a new Benchmark for Emotion Detection in NLP: A Unifying Framework of Recent Corpora

Emotion detection of social data: APIs comparative study

TorchEEGEMO: A deep learning toolbox towards EEG-based emotion recognition

A Gold Standard for Emotion Annotation in Stack Overflow

MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

MuSe-Toolbox: The Multimodal Sentiment Analysis Continuous Annotation Fusion and Discrete Class Transformation Toolbox

MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition

BioEmoDetector: A flexible platform for detecting emotions from health narratives

Emotimonitor: A Trello Power-Up to Capture Emotions of Agile Teams

Emotion Classification In Software Engineering Texts: A Comparative Analysis of Pre-trained Transformers Language Models

StockEmotions: Discover Investor Emotions for Financial Sentiment Analysis and Multivariate Time Series

Improving Emotion Analysis For Speech-Induced Eegs Through Eemd-Hht-Based Feature Extraction And Electrode Selection