Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Deep Learning-Based Image Segmentation on Multimodal Medical Imaging

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Multimodal Medical Imaging Using Modern Deep Learning Approaches

Self-Supervised Multi-Modal Hybrid Fusion Network for Brain Tumor Segmentation.

Medical image segmentation based on self-supervised hybrid fusion network

Learning Cross-Modal Deep Representations for Multi-Modal MR Image Segmentation

A review: Deep learning for medical image segmentation using multi-modality fusion

CM-SegNet: A Deep Learning-Based Automatic Segmentation Approach for Medical Images by Combining Convolution and Multilayer Perceptron

Deep Fusion of Shifted MLP and CNN for Medical Image Segmentation.

A multibranch and multiscale neural network based on semantic perception for multimodal medical image fusion

Effective Multipath Feature Extraction 3D CNN for Multimodal Brain Tumor Segmentation

An Attentive Multi-Modal CNN for Brain Tumor Radiogenomic Classification

Multi-modal Brain Image Segmentation Based on Multi-Encoder with Hybrid Lateral Connection (Mehlc-Net).

Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network

Multimodal Data-Driven Segmentation of Bone Metastasis Lesions in SPECT Bone Scans Using Deep Learning

Multimodal Fusion for Enhanced Semantic Segmentation in Brain Tumor Imaging: Integrating Deep Learning and Guided Filtering Via Advanced 3D Semantic Segmentation Architectures

A medical image segmentation method for rectal tumors based on multi‐scale feature retention and multiple attention mechanisms

SegCoFusion: an Integrative Multimodal Volumetric Segmentation Cooperating with Fusion Pipeline to Enhance Lesion Awareness.

A Novel Network Based on Densely Connected Fully Convolutional Networks for Segmentation of Lung Tumors on Multi-Modal MR Images

Feature interaction network based on hierarchical decoupled convolution for 3D medical image segmentation