Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Multi-vertebrae Segmentation from Arbitrary Spine MR Images Under Global View

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Multi-Scale Hybrid Attention Convolutional Neural Network for Automatic Segmentation of Lumbar Vertebrae From MRI

A Localisation-Segmentation Approach for Multi-label Annotation of Lumbar Vertebrae using Deep Nets

Spinal Vertebrae Segmentation and Localization by Transfer Learning

Multi-View Vertebra Localization and Identification from CT Images

WGAN-based multi-structure segmentation of vertebral cross-section MRI using ResU-Net and clustered transformer

MRLN: Multi-Task Relational Learning Network for MRI Vertebral Localization, Identification, and Segmentation

Vertebrae Labeling via End-to-End Integral Regression Localization and Multi-Label Classification Network

Multi-modal vertebrae recognition using Transformed Deep Convolution Network

Multi-task Contexture Learning Network for automated vertebrae segmentation and tumor diagnosis from MRI

Automatic Segmentation, Localization, and Identification of Vertebrae in 3D CT Images Using Cascaded Convolutional Neural Networks

Residual Block-based Multi-Label Classification and Localization Network with Integral Regression for Vertebrae Labeling

SpineMamba: Enhancing 3D Spinal Segmentation in Clinical Imaging through Residual Visual Mamba Layers and Shape Priors

Semi-Supervised Hybrid Spine Network for Segmentation of Spine MR Images

Regression Segmentation for $M^{3}$ Spinal Images

Sequence based local–global information fusion framework for vertebrae detection under pathological and FOV variation challenges

VertDetect: Fully End-to-End 3D Vertebral Instance Segmentation Model

Automatic Spine Segmentation using Convolutional Neural Network via Redundant Generation of Class Labels for 3D Spine Modeling

Quality of Diabetes Care: A cross-sectional observational study in Oman.

RAU-Net: U-Net network based on residual multi-scale fusion and attention skip layer for overall spine segmentation