Abstract:Emotion recognition plays an important role in human life and healthcare. The EEG has been extensively researched as an objective indicator of intense emotions. However, current existing methods lack sufficient analysis of shallow and deep EEG features. In addition, human emotions are complex and variable, making it difficult to comprehensively represent emotions using a single-modal signal. As a signal associated with gaze tracking and eye movement detection, Eye-related signals provide various forms of supplementary information for multimodal emotion analysis. Therefore, we propose a Pseudo-Siamese Pyramid Network (PSPN) for multimodal emotion analysis. The PSPN model employs a Depthwise Separable Convolutional Pyramid (DSCP) to extract and integrate intrinsic emotional features at various levels and scales from EEG signals. Simultaneously, we utilize a fully connected subnetwork to extract the external emotional features from eye-related signals. Finally, we introduce a Pseudo-Siamese network that integrates a flexible cross-modal dual-branch subnetwork to collaboratively utilize EEG emotional features and eye-related behavioral features, achieving consistency and complementarity in multimodal emotion recognition. For evaluation, we conducted experiments on the DEAP and SEED-IV public datasets. The experimental results demonstrate that multimodal fusion significantly improves the accuracy of emotion recognition compared to single-modal approaches. Our PSPN model achieved the best accuracy of 96.02% and 96.45% on the valence and arousal dimensions of the DEAP dataset, and 77.81% on the SEED-IV dataset, respectively. Our code link is: https://github.com/Yinyanyan003/PSPN.git.

Emotion-Aroused Human Behaviors Perception Using RNNPB

Unsupervised Recurrent Neural Network with Parametric Bias Framework for Human Emotion Recognition with Multimodal Sensor Data Fusion

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Self-attention Transfer Networks for Speech Emotion Recognition

EEG Emotion Recognition Based on Self-attention Dynamic Graph Neural Networks

Brain signal based human emotion analysis by circular back propagation and Deep Kohonen Neural Networks

Spatial-Temporal Recurrent Neural Network for Emotion Recognition

EEG-based emotion recognition using simple recurrent units network and ensemble learning

A Hierarchical Emotion Regulated Sensorimotor Model: Case Studies

EEG-Based Emotion Recognition Using Hierarchical Network with Subnetwork Nodes

A Novel Bi-Hemispheric Discrepancy Model for EEG Emotion Recognition

Long Short Term Memory Recurrent Neural Network Based Encoding Method for Emotion Recognition in Video.

Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method

Optimized recurrent neural network based brain emotion recognition technique

Emotion Recognition from EEG Signals Using Recurrent Neural Networks

BiERU: Bidirectional Emotional Recurrent Unit for Conversational Sentiment Analysis

A Parallel Neural Networks for Emotion Recognition Based on EEG Signals

PSPN: Pseudo-Siamese Pyramid Network for multimodal emotion analysis

EEG Emotion Recognition Employing RGPCN-BiGRUAM: ReliefF-Based Graph Pooling Convolutional Network and BiGRU Attention Mechanism

Video-based Emotion Recognition Using Multi-dichotomy RNN-DNN

Sentiment Recognition Model of EEG Signals Combined with One-Dimensional Convolution and BiBASRU-AT