Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Fusion Learning of Multimodal Neuroimaging with Weighted Graph AutoEncoder.

Regional Brain Fusion: Graph Convolutional Network for Alzheimer's Disease Prediction and Analysis

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

A Heterogeneous Graph Based Framework for Multimodal Neuroimaging Fusion Learning

Mapping Multi-Modal Brain Connectome for Brain Disorder Diagnosis Via Cross-Modal Mutual Learning.

Multi-modal Dynamic Graph Network: Coupling Structural and Functional Connectome for Disease Diagnosis and Classification

Joint Embedding of Structural and Functional Brain Networks with Graph Neural Networks for Mental Illness Diagnosis

Graph-Based Fusion of Imaging, Genetic and Clinical Data for Degenerative Disease Diagnosis.

Multi-graph Fusion for Functional Neuroimaging Biomarker Detection

Hybrid Multimodal Fusion for Graph Learning in Disease Prediction

Multi-modal imaging genetics data fusion by deep auto-encoder and self-representation network for Alzheimer's disease diagnosis and biomarkers extraction

MSTGC: Multi-Channel Spatio-Temporal Graph Convolution Network for Multi-Modal Brain Networks Fusion

IG-GRD: A Model Based on Disentangled Graph Representation Learning for Imaging Genetic Data Fusion

Fusing Structural and Functional Connectivities using Disentangled VAE for Detecting MCI

Hypercomplex Graph Neural Network: Towards Deep Intersection of Multi-modal Brain Networks

A Heterogeneous Graph Neural Network Fusing Functional and Structural Connectivity for MCI Diagnosis

Fusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction from Medical Data and Beyond

Multimodal Cross Enhanced Fusion Network for Diagnosis of Alzheimer's Disease and Subjective Memory Complaints.

Multimodal cross enhanced fusion network for diagnosis of Alzheimer?s disease and

Multi-modal Graph Neural Network for Early Diagnosis of Alzheimer's Disease from sMRI and PET Scans

Two-Dimensional Attentive Fusion for Multi-Modal Learning of Neuroimaging and Genomics Data