Abstract:Various physiological signals can reflect the human’s emotional states objectively. How to take advantage of the common as well as complementary properties of different physiological signals in representing the emotional states is an interesting problem. Although various models have been constructed to fuse multimodal physiological signals for emotion recognition, the possible incongruity existing among different physiological signals in representing the emotional states and the redundancy resulted from the fusion, which may affect the performance of the fusion schemes seriously, were seldom considered. To this end, a fusion model, which can eliminate the incongruity among different physiological signals and reduce the information redundancy to some extent, is proposed. First, one physiological signal is chosen as the primary modality due to its prominent performance in emotion recognition, and the remaining physiological signals are viewed as the auxiliary modalities. Secondly, the Cross Modal Transformer (CMT) is adopted to optimize the features of the auxiliary modalities by eliminating the incongruity among them, and then Low Rank Fusion (LRF) is performed to eliminate information redundancy caused by fusion. Thirdly, the modified CMT (MCMT) is constructed to enhance the feature of the primary modality by that of each optimized auxiliary modality feature. Fourthly, Self-Attention Transformer (SAT) is performed on the concatenation result of all the enhanced primary modality features to take full advantage of the common as well complementary properties among them in representing the emotional states. Finally, the enhanced primary modality feature and the optimized auxiliary features are fused by concatenation for emotion recognition. Extensive experimental results on DEAP and WESAD datasets demonstrate that (i) The incongruity does exist among different physiological signals, and the CMT-based auxiliary modality feature optimization strategy can eliminate the incongruity prominently; (ii) The emotion prediction accuracy of the primary modality can be enhanced by the auxiliary modality; (iii) All the key modules in the proposed model, CMT, LRF, and MCMT, contribute to the performance enhancement of the proposed model; iv) The proposed model outperforms State-Of-The-Art (SOTA) models in emotion recognition task.

GJFusion: A Channel-Level Correlation Construction Method for Multimodal Physiological Signal Fusion

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

A multi-stage dynamical fusion network for multimodal emotion recognition

Fusionatt: Deep Fusional Attention Networks For Multi-Channel Biomedical Signals

Incongruity-aware Multimodal Physiology Signals Fusion for Emotion Recognition

A Network-based Multimodal Data Fusion Approach for Characterizing Dynamic Multimodal Physiological Patterns

Feature-level fusion of multimodal physiological signals for emotion recognition

Multimodal Physiological Signals Fusion for Online Emotion Recognition

SFusion: Self-attention based N-to-One Multimodal Fusion Block

Hierarchical multimodal-fusion of physiological signals for emotion recognition with scenario adaption and contrastive alignment

Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals

GraphMFT: A Graph Network Based Multimodal Fusion Technique for Emotion Recognition in Conversation

Hierarchical fusion of visual and physiological signals for emotion recognition

Multi-modal fusion network with complementarity and importance for emotion recognition

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals

Emotion Recognition Based on Weighted Fusion Strategy of Multichannel Physiological Signals