Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Multi-Grained Cross-modal Alignment for Learning Open-vocabulary Semantic Segmentation from Text Supervision

Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training

Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignments

ASIMSA: Advanced Semantic Information Guided Multi-Scale Alignment Framework for Medical Vision-Language Pretraining

Cross-Modality Medical Image Segmentation via Enhanced Feature Alignment and Cross Pseudo Supervision Learning

Anatomical Structure-Guided Medical Vision-Language Pre-training

Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration

Mind the Gap: Alleviating Local Imbalance for Unsupervised Cross-Modality Medical Image Segmentation.

Eye Gaze Guided Cross-Modal Alignment Network for Radiology Report Generation.

Granularity Matters: Pathological Graph-driven Cross-modal Alignment for Brain CT Report Generation

A vision-language model with multi-granular knowledge fusion in medical imaging

MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training

Multifocal region-assisted cross-modality learning for chest X-ray report generation

LoGra-Med: Long Context Multi-Graph Alignment for Medical Vision-Language Model

Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation

Multi-modal vision-language model for generalizable annotation-free pathology localization and clinical diagnosis

Fine-Grained Image-Text Alignment in Medical Imaging Enables Cyclic Image-Report Generation

AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation