Abstract:Physiological signal-based ubiquitous computing has garnered significant attention. However, the heterogeneity among multimodal physiological signals poses a critical challenge to practical applications. To traverse this heterogeneity gap, recent studies have focused on establishing inter-modality correlations. Early works only consider coarse-level correlations between the embeddings of each modality. More recent graph-based approaches incorporate prior knowledge-based correlations, although they may not be entirely accurate. In this paper, we propose the Graph Joint Fusion (GJFusion) network, which leverages channel-level inter-modality correlations based on a graph joint to mitigate the heterogeneous gap. Our proposed GJFusion first represents each modality as a graph, with each vertex corresponding to a signal channel, and the edges denoting their functional connectivity. We then join each modality by constructing inter-modality correlations for each salient channel using a sampling-based matching method. Discarded channels are transformed into a virtual vertex through a lightweight pooling operation. Subsequently, the fusion network integrates intra- and inter-modality features, enabling multimodal physiological signal fusion. To validate the effectiveness of our method, we select emotional state recognition as the downstream task and conduct comprehensive experiments on two benchmark datasets. The results demonstrate that our proposed GJFusion network surpasses the latest state-of-the-art methods, achieving relative accuracy improvements of 1.22% and 0.81% on the DEAP and MAHNOB-HCI datasets, respectively. Furthermore, visualization experiments of the salient brain regions reveal the presence of interpretable knowledge within the proposed GJFusion model.

Similarity network fusion of multiview power system data

Adaptive Graph Fusion Learning for Multi-View Spectral Clustering

Evaluating Composite Approaches to Modelling High-Dimensional Stochastic Variables in Power Systems

Multimedia data fusion method based on wireless sensor network in intelligent transportation system

Multi-level information fusion to alleviate network congestion

A Fusion Framework Using Integrated Neural Network Model for Non-Intrusive Load Monitoring

Multi-Variate Data Fusion Technique for Reducing Sensor Errors in Intelligent Transportation Systems

A Network-based Multimodal Data Fusion Approach for Characterizing Dynamic Multimodal Physiological Patterns

A Bad Data Resilient Multisensor Fusion Framework for Hybrid State Estimation

Joint Learning of Latent Similarity and Local Embedding for Multi-View Clustering

Multi-scale Geometric Summaries for Similarity-based Sensor Fusion

Similarity network fusion for scholarly journals

GJFusion: A Channel-Level Correlation Construction Method for Multimodal Physiological Signal Fusion

Multi-graph Fusion for Multi-view Spectral Clustering

Multi-graph fusion based on circuit networks

Fine-grained similarity fusion for Multi-view Spectral Clustering

A Similarity Matrix Low-Rank Approximation and Inconsistency Separation Fusion Approach for Multiview Clustering.

Multisensor data fusion for computer network information discovery

SimFusion+: extending simfusion towards efficient estimation on large and dynamic networks.

Multivariate Exploration of Non-Intrusive Load Monitoring Via Spatiotemporal Pattern Network

Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects