Abstract:Introduction: EEG-based emotion recognition has gradually become a new research direction, known as affective Brain-Computer Interface (aBCI), which has huge application potential in human-computer interaction and neuroscience. However, how to extract spatio-temporal fusion features from complex EEG signals and build learning method with high recognition accuracy and strong interpretability is still challenging. Methods: In this paper, we propose a hybrid attention spatio-temporal feature fusion network for EEG-based emotion recognition. First, we designed a spatial attention feature extractor capable of merging shallow and deep features to extract spatial information and adaptively select crucial features under different emotional states. Then, the temporal feature extractor based on the multi-head attention mechanism is integrated to perform spatio-temporal feature fusion to achieve emotion recognition. Finally, we visualize the extracted spatial attention features using feature maps, further analyzing key channels corresponding to different emotions and subjects. Results: Our method outperforms the current state-of-the-art methods on two public datasets, SEED and DEAP. The recognition accuracy are 99.12% ± 1.25% (SEED), 98.93% ± 1.45% (DEAP-arousal), and 98.57% ± 2.60% (DEAP-valence). We also conduct ablation experiments, using statistical methods to analyze the impact of each module on the final result. The spatial attention features reveal that emotion-related neural patterns indeed exist, which is consistent with conclusions in the field of neurology. Discussion: The experimental results show that our method can effectively extract and fuse spatial and temporal information. It has excellent recognition performance, and also possesses strong robustness, performing stably across different datasets and experimental environments for emotion recognition.

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Emotion recognition from users' EEG signals with the help of stimulus VIDEOS

Hybrid video emotional tagging using users’ EEG and video content

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition

Inter-Brain EEG Feature Extraction and Analysis for Continuous Implicit Emotion Tagging During Video Watching

An Investigation of Olfactory-Enhanced Video on EEG-Based Emotion Recognition

Channel attention convolutional aggregation network based on video-level features for EEG emotion recognition

Multimodal Emotion Recognition Based on EEG and EOG Signals Evoked by the Video-Odor Stimuli

Multimodal emotion recognition using EEG and eye tracking data.

Emotion recognition with the help of privileged information

EEG-Based Emotion Recognition Fusing Spacial-Frequency Domain Features and Data-Driven Spectrogram-Like Features.

Multimodal Emotion Recognition Based on Feature Selection and Extreme Learning Machine in Video Clips.

Mutual Information-Based Emotion Recognition

[An improved electroencephalogram feature extraction algorithm and its application in emotion recognition]

Emotion Recognition from EEG Signals by Leveraging Stimulus Videos.

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

Fusion of Facial Expressions and EEG for Emotion Recognition

Multi-Target Positive Emotion Recognition from EEG Signals