Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Multi-Attention Mechanism Medical Image Segmentation Combined with Word Embedding Technology

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Segmentation of medical images using an attention embedded lightweight network

Collaborative multi-feature extraction and scale-aware semantic information mining for medical image segmentation

[Multi-scale medical image segmentation based on pixel encoding and spatial attention mechanism]

A medical image segmentation method for rectal tumors based on multi‐scale feature retention and multiple attention mechanisms

MSMHSA-DeepLab V3+: An Effective Multi-Scale, Multi-Head Self-Attention Network for Dual-Modality Cardiac Medical Image Segmentation

Multimodal parallel attention network for medical image segmentation

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

MCANet: Medical Image Segmentation with Multi-Scale Cross-Axis Attention

HybridCANet: UNet-like Network Bridging CNN and Attention for Multimodal Brain Image Segmentation.

Deep Learning-Based Image Segmentation on Multimodal Medical Imaging

MC-Net: multi-scale context-attention network for medical CT image segmentation

Automated cardiac segmentation of cross-modal medical images using unsupervised multi-domain adaptation and spatial neural attention structure

MFHARFNet: Multi-branch feature hybrid and adaptive receptive field network for image segmentation

CASF-Net: Cross-attention and Cross-scale Fusion Network for Medical Image Segmentation

MCPA: Multi-scale Cross Perceptron Attention Network for 2D Medical Image Segmentation

Multi-Receptive-Field CNN for Semantic Segmentation of Medical Images

Fusion of medical imaging and electronic health records with attention and multi-head machanisms

MACCoM: A multiple attention and convolutional cross-mixer framework for detailed 2D biomedical image segmentation

MTANet: Multi-Task Attention Network for Automatic Medical Image Segmentation and Classification