Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Convolutional neural network with coarse-to-fine resolution fusion and residual learning structures for cross-modality image synthesis

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Improved UNet‐based Magnetic Resonance Imaging Segmentation of Demyelinating Diseases with Small Lesion Regions

A Novel Method Of Synthetic Ct Generation From Mr Images Based On Convolutional Neural Networks

Learning Unified Hyper-Network for Multi-Modal MR Image Synthesis and Tumor Segmentation With Missing Modalities

A layer-wise fusion network incorporating self-supervised learning for multimodal MR image synthesis

Multi-Modality MR Image Synthesis via Confidence-Guided Aggregation and Cross-Modality Refinement

Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis

Simultaneous Super-Resolution and Cross-Modality Synthesis of 3D Medical Images using Weakly-Supervised Joint Convolutional Sparse Coding

An hetero-modal deep learning framework for medical image synthesis applied to contrast and non-contrast MRI

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Unified Multi-Modal Image Synthesis for Missing Modality Imputation

Enhancing CT Image synthesis from multi-modal MRI data based on a multi-task neural network framework

Three-Dimensional Medical Image Fusion with Deformable Cross-Attention

Uni-COAL: A Unified Framework for Cross-Modality Synthesis and Super-Resolution of MR Images

Cross-Modality Neuroimage Synthesis: A Survey

A weakly supervised model for incomplete multimodal MRI synthesis with tumor‐aware approach

Learning Cross-Modal Deep Representations for Multi-Modal MR Image Segmentation

Bidirectional feature matching based on deep pairwise contrastive learning for multiparametric MRI image synthesis

CT synthesis from MRI with an improved multi-scale learning network