Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Robust Divergence Learning for Missing-Modality Segmentation

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Generative Learning-Based Lightweight MRI Brain Tumor Segmentation with Missing Modalities

Modality-Specific Information Disentanglement from Multi-parametric MRI for Breast Tumor Segmentation and Computer-aided Diagnosis

Multimodality-Assisted Semi-Supervised Brain Tumor Segmentation in Nondominant Modality Based on Consistency Learning

Brain Tumor Segmentation on MRI with Missing Modalities

Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning

Disentangle, align and fuse for multimodal and semi-supervised image segmentation

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

Deformation-aware and reconstruction-driven multimodal representation learning for brain tumor segmentation with missing modalities

Medical Image Segmentation on MRI Images with Missing Modalities: A Review

DIGEST: Deeply supervIsed knowledGE tranSfer neTwork learning for brain tumor segmentation with incomplete multi-modal MRI scans

Decoupling Feature Representations of Ego and Other Modalities for Incomplete Multi-modal Brain Tumor Segmentation

Modality-Pairing Learning for Brain Tumor Segmentation

Pre-Post Interaction Learning for Brain Tumor Segmentation with Missing MRI Modalities.

Learning Unified Hyper-Network for Multi-Modal MR Image Synthesis and Tumor Segmentation With Missing Modalities

Uncertainty-Aware Multi-Dimensional Mutual Learning for Brain and Brain Tumor Segmentation

HeMIS: Hetero-Modal Image Segmentation

Adapting Segment Anything Model for 3D Brain Tumor Segmentation With Missing Modalities