Abstract:Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.

Emotion Recognition Based on a EEG–fNIRS Hybrid Brain Network in the Source Space

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Cross-Subject Emotion Recognition Brain–Computer Interface Based on fNIRS and DBJNet

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

EEG-Based Emotion Recognition by Exploiting Fused Network Entropy Measures of Complex Networks Across Subjects.

Improving EEG-based Emotion Recognition by Fusing Time-frequency And Spatial Representations

Investigating EEG-based functional connectivity patterns for multimodal emotion recognition

EEG-based Emotion Recognition with Feature Fusion Networks.

Study on multidimensional emotion recognition fusing dynamic brain network features in EEG signals

Emotion Recognition from Multiple Physiological Signals Using Intra- and Inter-Modality Attention Fusion Network

Spatial-temporal Network for Fine-grained-level Emotion EEG Recognition

An Attention-Based Multi-Domain Bi-Hemisphere Discrepancy Feature Fusion Model for EEG Emotion Recognition

EEGFuseNet: Hybrid Unsupervised Deep Feature Characterization and Fusion for High-Dimensional EEG with An Application to Emotion Recognition

A Feature-Fused Convolutional Neural Network for Emotion Recognition from Multichannel EEG Signals

Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition

EEG-Based Emotion Recognition Fusing Spacial-Frequency Domain Features and Data-Driven Spectrogram-Like Features.

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition.

Adaptive Dual-Space Network with Multigraph Fusion for EEG-Based Emotion Recognition

Fusing Frequency-Domain Features and Brain Connectivity Features for Cross-Subject Emotion Recognition

Horizontal and Vertical Features Fusion Network Based on Different Brain Regions for Emotion Recognition