Abstract:Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.

MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

Using electroencephalogram for emotion recognition based on filter-bank long short-term memory networks

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

EmotionMeter: A Multimodal Framework for Recognizing Human Emotions.

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition

MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

Multimodal Emotion Recognition Model using Physiological Signals

Multi-modal emotion recognition using EEG and speech signals

Emotion Recognition Using Cross-Modal Attention from Eeg and Facial Expression

Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Multidimensional Feature in Emotion Recognition Based on Multi-Channel EEG Signals.

Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism

Multi-Modal Emotion Recognition Based On deep Learning Of EEG And Audio Signals

Emotion recognition framework using multiple modalities for an effective human–computer interaction