Abstract:Physiological signals can more realistically reflect human emotional states. To overcome the limitations imposed in single-modal emotion recognition, emotion recognition of multimodal physiological signals has received increasingly widespread attention. However, the original fusion models usually ignore the different distributions of multiple signals and how to capture complementary features from multimodal information effectively. This paper proposes an effective classification model for multimodal physiological signals to address the above issues based on modeling the heterogeneity and correlation between multimodal signals. First, differential entropy features are extracted from Electroencephalography (EEG) signals and peripheral physiological signals (PPS) such as Electrocardiographic (ECG) signals, Electromyographic (EMG) signals, and other physiological signals. Then, according to the different distributions and frequency characteristics of the acquired signals, the EEG signal features are made into a three-dimensional feature map and input to the neural network to extract the frequency spatial dimension features. Further temporal features are extracted from the peripheral physiological signals using a long and short-term memory network. Finally, the EEG and peripheral physiological signal features were fused and input to a multimodal long and short-term memory network to extract the association between different modalities and perform classification. The experiments were conducted on the benchmark DEAP dataset, and the results showed that the classification accuracy of the proposed model in this paper was 95.89% and 94.99% in the arousal dimension and the valence dimension, respectively, which were 2.77% and 3.11% higher compared to the unimodal EEG model, respectively. This paper also analyzed the effects of different peripheral physiological signals on emotion recognition.

Feature-level fusion of multimodal physiological signals for emotion recognition

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Emotion Recognition in Videos via Fusing Multimodal Features.

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Emotion recognition based on multiple physiological signals

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals

Multimodal Emotion Recognition Model using Physiological Signals

Multimodal Emotion Recognition by Combining Physiological Signals and Facial Expressions: a Preliminary Study.

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Emotion Recognition Based on Weighted Fusion Strategy of Multichannel Physiological Signals

A multi-stage dynamical fusion network for multimodal emotion recognition

Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals

Hierarchical fusion of visual and physiological signals for emotion recognition

Hierarchical multimodal-fusion of physiological signals for emotion recognition with scenario adaption and contrastive alignment

Multimodal Emotion Recognition Using Different Fusion Techniques

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data