Abstract:Objective.The study of emotion recognition through electroencephalography (EEG) has garnered significant attention recently. Integrating EEG with other peripheral physiological signals may greatly enhance performance in emotion recognition. Nonetheless, existing approaches still suffer from two predominant challenges: modality heterogeneity, stemming from the diverse mechanisms across modalities, and fusion credibility, which arises when one or multiple modalities fail to provide highly credible signals.Approach.In this paper, we introduce a novel multimodal physiological signal fusion model that incorporates both intra-inter modality reconstruction and sequential pattern consistency, thereby ensuring a computable and credible EEG-based multimodal emotion recognition. For the modality heterogeneity issue, we first implement a local self-attention transformer to obtain intra-modal features for each respective modality. Subsequently, we devise a pairwise cross-attention transformer to reveal the inter-modal correlations among different modalities, thereby rendering different modalities compatible and diminishing the heterogeneity concern. For the fusion credibility issue, we introduce the concept of sequential pattern consistency to measure whether different modalities evolve in a consistent way. Specifically, we propose to measure the varying trends of different modalities, and compute the inter-modality consistency scores to ascertain fusion credibility.Main results.We conduct extensive experiments on two benchmarked datasets (DEAP and MAHNOB-HCI) with the subject-dependent paradigm. For the DEAP dataset, our method improves the accuracy by 4.58%, and the F1 score by 0.63%, compared to the state-of-the-art baseline. Similarly, for the MAHNOB-HCI dataset, our method improves the accuracy by 3.97%, and the F1 score by 4.21%. In addition, we gain much insight into the proposed framework through significance test, ablation experiments, confusion matrices and hyperparameter analysis. Consequently, we demonstrate the effectiveness of the proposed credibility modelling through statistical analysis and carefully designed experiments.Significance.All experimental results demonstrate the effectiveness of our proposed architecture and indicate that credibility modelling is essential for multimodal emotion recognition.

Hierarchical multimodal-fusion of physiological signals for emotion recognition with scenario adaption and contrastive alignment

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Hierarchical fusion of visual and physiological signals for emotion recognition

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Feature-level fusion of multimodal physiological signals for emotion recognition

Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism

Hierarchical Hypercomplex Network for Multimodal Emotion Recognition

Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals

A multi-stage dynamical fusion network for multimodal emotion recognition

Multimodal Emotion Recognition Model using Physiological Signals

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Emotion recognition based on multiple physiological signals

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Cross-modal credibility modelling for EEG-based multimodal emotion recognition

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

Fusion with Hierarchical Graphs for Mulitmodal Emotion Recognition