Abstract:Objective.The study of emotion recognition through electroencephalography (EEG) has garnered significant attention recently. Integrating EEG with other peripheral physiological signals may greatly enhance performance in emotion recognition. Nonetheless, existing approaches still suffer from two predominant challenges: modality heterogeneity, stemming from the diverse mechanisms across modalities, and fusion credibility, which arises when one or multiple modalities fail to provide highly credible signals.Approach.In this paper, we introduce a novel multimodal physiological signal fusion model that incorporates both intra-inter modality reconstruction and sequential pattern consistency, thereby ensuring a computable and credible EEG-based multimodal emotion recognition. For the modality heterogeneity issue, we first implement a local self-attention transformer to obtain intra-modal features for each respective modality. Subsequently, we devise a pairwise cross-attention transformer to reveal the inter-modal correlations among different modalities, thereby rendering different modalities compatible and diminishing the heterogeneity concern. For the fusion credibility issue, we introduce the concept of sequential pattern consistency to measure whether different modalities evolve in a consistent way. Specifically, we propose to measure the varying trends of different modalities, and compute the inter-modality consistency scores to ascertain fusion credibility.Main results.We conduct extensive experiments on two benchmarked datasets (DEAP and MAHNOB-HCI) with the subject-dependent paradigm. For the DEAP dataset, our method improves the accuracy by 4.58%, and the F1 score by 0.63%, compared to the state-of-the-art baseline. Similarly, for the MAHNOB-HCI dataset, our method improves the accuracy by 3.97%, and the F1 score by 4.21%. In addition, we gain much insight into the proposed framework through significance test, ablation experiments, confusion matrices and hyperparameter analysis. Consequently, we demonstrate the effectiveness of the proposed credibility modelling through statistical analysis and carefully designed experiments.Significance.All experimental results demonstrate the effectiveness of our proposed architecture and indicate that credibility modelling is essential for multimodal emotion recognition.

CiABL: Completeness-induced Adaptative Broad Learning for Cross-Subject Emotion Recognition with EEG and Eye Movement Signals

Multi-view Domain-Adaptive Representation Learning for EEG-based Emotion Recognition

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

A Novel Multi-Source Contrastive Learning Approach for Robust Cross-Subject Emotion Recognition in Eeg Data

Cross-Cultural Emotion Recognition with EEG and Eye Movement Signals Based on Multiple Stacked Broad Learning System

A Cross-subject and Cross-modal Model for Multimodal Emotion Recognition

Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition

EEG-Based Cross-Subject Emotion Recognition Using Sparse Bayesian Learning with Enhanced Covariance Alignment

Plug-and-Play Domain Adaptation for Cross-Subject EEG-based Emotion Recognition

An Attention-Enhanced Retentive Broad Learning System for Subject-Generic Emotion Recognition from EEG Signals.

Preliminary version do not cite

Cross-modal credibility modelling for EEG-based multimodal emotion recognition

FMLAN: A novel framework for cross-subject and cross-session EEG emotion recognition

Naturalistic Emotion Recognition Using EEG and Eye Movements

Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism

Functional Connectivity Patterns Learning for EEG-based Emotion Recognition

Multi-source Information-Shared Domain Adaptation for EEG Emotion Recognition.

Multi-Channel EEG Based Emotion Recognition Using Temporal Convolutional Network and Broad Learning System

Emotion recognition based on multiple physiological signals

Transfer Components Between Subjects For Eeg-Based Emotion Recognition

Exploring EEG Features in Cross-Subject Emotion Recognition