Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Multimodal Priors Guided Segmentation of Liver Lesions in MRI Using Mutual Information Based Graph Co-Attention Networks.

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Liver Tumor Detection Via A Multi-Scale Intermediate Multi-Modal Fusion Network on MRI Images.

Segmentation and Detection of Liver Tumor in Non-contrast MRI Image Combined with Multi-modality Features

Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Tumor Segmentation in Weakly Paired Anatomical and Functional MRI Images with Multimodal Information Fusion

Self-Supervised Multi-Modal Hybrid Fusion Network for Brain Tumor Segmentation.

Fully Convolutional DenseNet with Attention Mechanism for Liver Lesion Segmentation in MRI Images.

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion

Multi-Modal Co-Learning for Liver Lesion Segmentation on PET-CT Images

Context Fusion Network with Multi-Scale-aware Skip Connection and Twin-Split Attention for Liver Tumor Segmentation

A three-path network with multi-scale selective feature fusion, edge-inspiring and edge-guiding for liver tumor segmentation

Segmentation of Low-Grade Brain Tumors Using Mutual Attention Multimodal MRI

Coco-Attention for Tumor Segmentation in Weakly Paired Multimodal MRI Images

Feature fusion and latent feature learning guided brain tumor segmentation and missing modality recovery network

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

Brain Tumor Segmentation via Multi-Modalities Interactive Feature Learning

Effective Multipath Feature Extraction 3D CNN for Multimodal Brain Tumor Segmentation

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

AHCNet: an Application of Attention Mechanism and Hybrid Connection for Liver Tumor Segmentation in CT Volumes.