Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Attention in Attention for PET-CT Modality Consensus Lung Tumor Segmentation

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation

ISA-Net: Improved spatial attention network for PET-CT tumor segmentation

MRLA-Net: A Tumor Segmentation Network Embedded with a Multiple Receptive-Field Lesion Attention Module in PET-CT Images.

A Spatial Squeeze and Multimodal Feature Fusion Attention Network for Multiple Tumor Segmentation from PET–CT Volumes

PET-Guided Attention Network for Segmentation of Lung Tumors from PET/CT Images

ASE-Net: A tumor segmentation method based on image pseudo enhancement and adaptive-scale attention supervision module

Multimodality Attention-Guided 3-D Detection of Nonsmall Cell Lung Cancer in 18 F-FDG PET/CT Images

MSRA-Net: Tumor segmentation network based on Multi-scale Residual Attention

PA-Net: A phase attention network fusing venous and arterial phase features of CT images for liver tumor segmentation

MMCA-NET: A Multimodal Cross Attention Transformer Network for Nasopharyngeal Carcinoma Tumor Segmentation Based on a Total-Body PET/CT System

Cross Modality Fusion for Modality-Specific Lung Tumor Segmentation in PET-CT Images.

LSAM: L2-norm self-attention and latent space feature interaction for automatic 3D multi-modal head and neck tumor segmentation

H2ASeg: Hierarchical Adaptive Interaction and Weighting Network for Tumor Segmentation in PET/CT Images

MAU-Net: Multiple Attention 3D U-Net for Lung Cancer Segmentation on CT Images

Hybrid Attention Fusion Segmentation Network for Diffuse Large B-cell Lymphoma in PET-CT

Fine-Grained Lung Cancer Classification from PET and CT Images Based on Multidimensional Attention Mechanism

Multilevel Attention Unet Segmentation Algorithm for Lung Cancer Based on CT Images

Convolutional bi-directional learning and spatial enhanced attentions for lung tumor segmentation

Joint segmentation of tumors in 3D PET-CT images with a network fusing multi-view and multi-modal information