Abstract:Introduction: EEG-based emotion recognition has gradually become a new research direction, known as affective Brain-Computer Interface (aBCI), which has huge application potential in human-computer interaction and neuroscience. However, how to extract spatio-temporal fusion features from complex EEG signals and build learning method with high recognition accuracy and strong interpretability is still challenging. Methods: In this paper, we propose a hybrid attention spatio-temporal feature fusion network for EEG-based emotion recognition. First, we designed a spatial attention feature extractor capable of merging shallow and deep features to extract spatial information and adaptively select crucial features under different emotional states. Then, the temporal feature extractor based on the multi-head attention mechanism is integrated to perform spatio-temporal feature fusion to achieve emotion recognition. Finally, we visualize the extracted spatial attention features using feature maps, further analyzing key channels corresponding to different emotions and subjects. Results: Our method outperforms the current state-of-the-art methods on two public datasets, SEED and DEAP. The recognition accuracy are 99.12% ± 1.25% (SEED), 98.93% ± 1.45% (DEAP-arousal), and 98.57% ± 2.60% (DEAP-valence). We also conduct ablation experiments, using statistical methods to analyze the impact of each module on the final result. The spatial attention features reveal that emotion-related neural patterns indeed exist, which is consistent with conclusions in the field of neurology. Discussion: The experimental results show that our method can effectively extract and fuse spatial and temporal information. It has excellent recognition performance, and also possesses strong robustness, performing stably across different datasets and experimental environments for emotion recognition.

Spectrum-based channel attention cooperating with time continuity encoding in transformer for EEG emotion analysis

JDAT: Joint-Dimension-Aware Transformer with Strong Flexibility for EEG Emotion Recognition

Temporal Relative Transformer Encoding Cooperating with Channel Attention for EEG Emotion Analysis

Spatial-temporal Transformers for EEG Emotion Recognition

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

Temporal-spatial Representation Learning Transformer for EEG-based Emotion Recognition

Using electroencephalogram for emotion recognition based on filter-bank long short-term memory networks

AMDET: Attention based Multiple Dimensions EEG Transformer for Emotion Recognition

A spatiotemporal symmetrical transformer structure for EEG emotion recognition

Emotion Classification Based on Transformer and CNN for EEG Spatial–Temporal Feature Learning

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

Temporal aware Mixed Attention-based Convolution and Transformer Network for cross-subject EEG emotion recognition

EmT: A Novel Transformer for Generalized Cross-subject EEG Emotion Recognition

A Transformer Convolutional Network With the Method of Image Segmentation for EEG-Based Emotion Recognition

Channel attention convolutional aggregation network based on video-level features for EEG emotion recognition

EEG emotion recognition using attention-based convolutional transformer neural network

TSception: Capturing Temporal Dynamics and Spatial Asymmetry from EEG for Emotion Recognition

SST-CRAM: spatial-spectral-temporal based convolutional recurrent neural network with lightweight attention mechanism for EEG emotion recognition

S3T-Net: A novel electroencephalogram signals-oriented emotion recognition model

EEG Emotion Recognition Based on Frequency and Channel Convolutional Attention

Emotion Transformer Fusion - Complementary Representation Properties of EEG and Eye Movements on Recognizing Anger and Surprise.