Abstract:Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.

Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze

Emotion Recognition in Videos via Fusing Multimodal Features.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

A multimodal fusion-based deep learning framework combined with local-global contextual TCNs for continuous emotion recognition from videos

End-to-End Continuous Emotion Recognition from Video Using 3D Convlstm Networks

Emotion Recognition Using Cross-Modal Attention from Eeg and Facial Expression

Facial Expression and EEG Fusion for Investigating Continuous Emotions of Deaf Subjects

A Two-Stage Spatiotemporal Attention Convolution Network for Continuous Dimensional Emotion Recognition from Facial Video

Monocular 3D Facial Expression Features for Continuous Affect Recognition

Facial Expression Recognition in the Wild Based on Multimodal Texture Features

Emotion Recognition from Large-Scale Video Clips with Cross-Attention and Hybrid Feature Weighting Neural Networks

Learning Expression Features via Deep Residual Attention Networks for Facial Expression Recognition From Video Sequences

Learning facial expression and body gesture visual information for video emotion recognition

Exploring Multimodal Visual Features For Continuous Affect Recognition

Mutilmodal Feature Extraction and Attention-based Fusion for Emotion Estimation in Videos

Combining Eye Movements and EEG to Enhance Emotion Recognition.

Leveraging the Deep Learning Paradigm for Continuous Affect Estimation from Facial Expressions

Video facial emotion recognition based on local enhanced motion history image and CNN-CTSLSTM networks

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues

User Emotion Recognition Method Based on Facial Expression and Speech Signal Fusion

Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition