Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition

Xinke Shen,Xianggen Liu,Xin Hu,Dan Zhang,Sen Song

DOI: https://doi.org/10.1109/TAFFC.2022.3164516

2022-04-06

Abstract:EEG signals have been reported to be informative and reliable for emotion recognition in recent years. However, the inter-subject variability of emotion-related EEG signals still poses a great challenge for the practical applications of EEG-based emotion recognition. Inspired by recent neuroscience studies on inter-subject correlation, we proposed a Contrastive Learning method for Inter-Subject Alignment (CLISA) to tackle the cross-subject emotion recognition problem. Contrastive learning was employed to minimize the inter-subject differences by maximizing the similarity in EEG signal representations across subjects when they received the same emotional stimuli in contrast to different ones. Specifically, a convolutional neural network was applied to learn inter-subject aligned spatiotemporal representations from EEG time series in contrastive learning. The aligned representations were subsequently used to extract differential entropy features for emotion classification. CLISA achieved state-of-the-art cross-subject emotion recognition performance on our THU-EP dataset with 80 subjects and the publicly available SEED dataset with 15 subjects. It could generalize to unseen subjects or unseen emotional stimuli in testing. Furthermore, the spatiotemporal representations learned by CLISA could provide insights into the neural mechanisms of human emotion processing.

Human-Computer Interaction,Machine Learning,Signal Processing,Neurons and Cognition

What problem does this paper attempt to address?

The paper attempts to address the issue of how to overcome the variability of EEG signals between different individuals in cross-subject emotion recognition to improve the accuracy and generalization ability of emotion recognition. Specifically, researchers have found that although EEG signals are rich and reliable in emotion recognition, there is significant variability in emotion-related EEG signals between different individuals, which poses a huge challenge for practical applications. The presence of this variability leads to the performance of cross-subject emotion recognition being far lower than that of single-subject recognition. To solve this problem, this paper proposes a contrastive learning-based method called Cross-Subject Aligned Contrastive Learning (CLISA). This method reduces the differences between different subjects by maximizing the similarity between EEG signal representations under the same emotional stimuli while minimizing the similarity between signal representations under different emotional stimuli. Specifically, CLISA uses Convolutional Neural Networks (CNN) to learn cross-subject aligned spatio-temporal representations from EEG time series and further extracts differential entropy features for emotion classification. Through this method, CLISA not only improves the performance of cross-subject emotion recognition but also generalizes to unseen subjects or unseen emotional stimuli during testing. Additionally, the spatio-temporal representations learned by CLISA can provide insights into the neural mechanisms of human emotion processing.

Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition

A Novel Multi-Source Contrastive Learning Approach for Robust Cross-Subject Emotion Recognition in Eeg Data

Emotion recognition of EEG signals based on contrastive learning graph convolutional model

A Novel Transfer Learning Model for Cross-Subject Emotion Recognition Using EEGs

CLDTA: Contrastive Learning based on Diagonal Transformer Autoencoder for Cross-Dataset EEG Emotion Recognition

Contrastive Learning of Shared Spatiotemporal EEG Representations Across Individuals for Naturalistic Neuroscience

Exploring EEG Features in Cross-Subject Emotion Recognition

CiABL: Completeness-induced Adaptative Broad Learning for Cross-Subject Emotion Recognition with EEG and Eye Movement Signals

Plug-and-Play Domain Adaptation for Cross-Subject EEG-based Emotion Recognition

GC-STCL: A Granger Causality-Based Spatial–Temporal Contrastive Learning Framework for EEG Emotion Recognition

Semi-Supervised Dual-Stream Self-Attentive Adversarial Graph Contrastive Learning for Cross-Subject EEG-based Emotion Recognition

EEG-Based Cross-Subject Emotion Recognition Using Sparse Bayesian Learning with Enhanced Covariance Alignment

Cross-Cultural Emotion Recognition with EEG and Eye Movement Signals Based on Multiple Stacked Broad Learning System

EEG-based cross-subject emotion recognition using multi-source domain transfer learning

Cascaded Self-supervised Learning for Subject-independent EEG-based Emotion Recognition

EEG-Based Subject-Independent Emotion Recognition Using Gated Recurrent Unit and Minimum Class Confusion

Cross-subject EEG Emotion Recognition Combined with Connectivity Features and Meta-Transfer Learning

Multi-subject Subspace Alignment for Non-Stationary EEG-based Emotion Recognition

Joint Contrastive Learning with Feature Alignment for Cross-Corpus EEG-based Emotion Recognition

EEG-SCMM: Soft Contrastive Masked Modeling for Cross-Corpus EEG-Based Emotion Recognition

Cross-Subject Eeg-Based Emotion Recognition With Deep Domain Confusion