Abstract:3D medical image analysis is pivotal in numerous clinical applications. However, the scarcity of labeled data and limited generalization capabilities hinder the advancement of AI-empowered models. Radiology reports are easily accessible and can serve as weakly-supervised signals. However, large-scale vision-language pre-training (VLP) remains underexplored in 3D medical image analysis. Specifically, the insufficient investigation into multi-grained radiology semantics and their correlations across patients leads to underutilization of large-scale volume-report data. Considering intra-patient cross-modal semantic consistency and inter-patient semantic correlations, we propose a multi-task VLP method, MG-3D, pre-trained on large-scale data (47.1K), addressing the challenges by the following two aspects: 1) Establishing the correspondence between volume semantics and multi-grained medical knowledge of each patient with cross-modal global alignment and complementary modality-guided local reconstruction, ensuring intra-patient features of different modalities cohesively represent the same semantic content; 2) Correlating inter-patient visual semantics based on fine-grained report correlations across patients, and keeping sensitivity to global individual differences via contrastive learning, enhancing the discriminative feature representation. Furthermore, we delve into the scaling law to explore potential performance improvements. Comprehensive evaluations across nine uni- and cross-modal clinical tasks are carried out to assess model efficacy. Extensive experiments on both internal and external datasets demonstrate the superior transferability, scalability, and generalization of MG-3D, showcasing its potential in advancing feature representation for 3D medical image analysis. Code will be available: <a class="link-external link-https" href="https://github.com/Xuefeng-Ni/MG-3D" rel="external noopener nofollow">this https URL</a>.

Autoregressive Sequence Modeling for 3D Medical Image Representation

MsVRL: Self-Supervised Multiscale Visual Representation Learning Via Cross-Level Consistency for Medical Image Segmentation

3D Self-Supervised Methods for Medical Imaging

Synthesis-based imaging-differentiation representation learning for multi-sequence 3D/4D MRI

Whole Heart 3D+T Representation Learning Through Sparse 2D Cardiac MR Images

C V ] 5 O ct 2 01 9 Self-supervised Feature Learning for 3 D Medical Images by Playing a Rubik ’ s Cube

Representation Recovering for Self-Supervised Pre-training on Medical Images.

A General Global and Local Pre-Training Framework for 3D Medical Image Segmentation.

Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cube

Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-training

Multiview Long-Short Spatial Contrastive Learning For 3D Medical Image Analysis

GMIM: Self-supervised pre-training for 3D medical image segmentation with adaptive and hierarchical masked image modeling

Bootstrap Representation Learning for Segmentation on Medical Volumes and Sequences

Semi-MedSeq: Semi-supervised Semantic Segmentation for Medical Image Sequences.

Re-weighting and Hierarchical Pre-training Boost 3D Medical Self-Supervised Representation.

Reinforcement Learning Driven Intra-modal and Inter-modal Representation Learning for 3D Medical Image Classification

Time-to-Event Pretraining for 3D Medical Imaging

MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training

Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training

Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis

Learning Brain Tumor Representation in 3D High-Resolution MR Images via Interpretable State Space Models