Abstract:Various physiological signals can reflect the human’s emotional states objectively. How to take advantage of the common as well as complementary properties of different physiological signals in representing the emotional states is an interesting problem. Although various models have been constructed to fuse multimodal physiological signals for emotion recognition, the possible incongruity existing among different physiological signals in representing the emotional states and the redundancy resulted from the fusion, which may affect the performance of the fusion schemes seriously, were seldom considered. To this end, a fusion model, which can eliminate the incongruity among different physiological signals and reduce the information redundancy to some extent, is proposed. First, one physiological signal is chosen as the primary modality due to its prominent performance in emotion recognition, and the remaining physiological signals are viewed as the auxiliary modalities. Secondly, the Cross Modal Transformer (CMT) is adopted to optimize the features of the auxiliary modalities by eliminating the incongruity among them, and then Low Rank Fusion (LRF) is performed to eliminate information redundancy caused by fusion. Thirdly, the modified CMT (MCMT) is constructed to enhance the feature of the primary modality by that of each optimized auxiliary modality feature. Fourthly, Self-Attention Transformer (SAT) is performed on the concatenation result of all the enhanced primary modality features to take full advantage of the common as well complementary properties among them in representing the emotional states. Finally, the enhanced primary modality feature and the optimized auxiliary features are fused by concatenation for emotion recognition. Extensive experimental results on DEAP and WESAD datasets demonstrate that (i) The incongruity does exist among different physiological signals, and the CMT-based auxiliary modality feature optimization strategy can eliminate the incongruity prominently; (ii) The emotion prediction accuracy of the primary modality can be enhanced by the auxiliary modality; (iii) All the key modules in the proposed model, CMT, LRF, and MCMT, contribute to the performance enhancement of the proposed model; iv) The proposed model outperforms State-Of-The-Art (SOTA) models in emotion recognition task.

A One-Step Physiological Status Assessment Method Fusing Subject-Variant Information

A Physiological Status Diagnosis Method Using Tensor-Based Regularization.

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Incongruity-aware Multimodal Physiology Signals Fusion for Emotion Recognition

Transformer-Based Multi-Modal Data Fusion Method for COPD Classification and Physiological and Biochemical Indicators Identification

Canonical Polyadic Decomposition with Auxiliary Information for Brain-Computer Interface.

Robust coupled tensor decomposition and feature extraction for multimodal medical data

Summit Vitals: Multi-Camera and Multi-Signal Biosensing at High Altitudes

Online MECG Compression Based on Incremental Tensor Decomposition for Wearable Devices

Coupled CP tensor decomposition with shared and distinct components for multi-task fMRI data fusion

Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation

Shift-invariant rank-( L , L , 1, 1) BTD with 3D spatial pooling and orthonormalization: Application to multi-subject fMRI data

Multi-Paradigm fMRI Fusion via Sparse Tensor Decomposition in Brain Functional Connectivity Study

Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling

Mdmnet: Multi-Dimensional Multi-Modal Network to Identify Organ System Limitation in Cardiopulmonary Exercise Testing

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

Multi-Scale and Multi-Channel Information Fusion for Exercise Electrocardiogram Feature Extraction and Classification

A Lightweight Hybrid Model Using Multiscale Markov Transition Field for Real-Time Quality Assessment of Photoplethysmography Signals

TensorBeat: Tensor Decomposition for Monitoring Multi-Person Breathing Beats with Commodity WiFi

GJFusion: A Channel-Level Correlation Construction Method for Multimodal Physiological Signal Fusion

Analytical fuzzy approach to biological data analysis.