Abstract:The integration of deep learning systems into healthcare has been hindered by the resource-intensive process of data annotation and the inability of these systems to generalize to different data distributions. Foundation models, which are models pre-trained on large datasets, have emerged as a solution to reduce reliance on annotated data and enhance model generalizability and robustness. DINOv2 is an open-source foundation model pre-trained with self-supervised learning on 142 million curated natural images that exhibits promising capabilities across various vision tasks. Nevertheless, a critical question remains unanswered regarding DINOv2's adaptability to radiological imaging, and whether its features are sufficiently general to benefit radiology image analysis. Therefore, this study comprehensively evaluates the performance DINOv2 for radiology, conducting over 200 evaluations across diverse modalities (X-ray, CT, and MRI). To measure the effectiveness and generalizability of DINOv2's feature representations, we analyze the model across medical image analysis tasks including disease classification and organ segmentation on both 2D and 3D images, and under different settings like kNN, few-shot learning, linear-probing, end-to-end fine-tuning, and parameter-efficient fine-tuning. Comparative analyses with established supervised, self-supervised, and weakly-supervised models reveal DINOv2's superior performance and cross-task generalizability. The findings contribute insights to potential avenues for optimizing pre-training strategies for medical imaging and enhancing the broader understanding of DINOv2's role in bridging the gap between natural and radiological image analysis. Our code is available at <a class="link-external link-https" href="https://github.com/MohammedSB/DINOv2ForRadiology" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in medical image analysis, existing deep - learning systems rely on a large amount of labeled data and have insufficient generalization ability across different data distributions. To reduce the dependence on labeled data and improve the generalization ability and robustness of the model, researchers have proposed Foundation Models (FMs). These models are usually pre - trained on large - scale datasets to achieve generalization ability across tasks and domains. Specifically in this paper, the researchers focus on the adaptability of DINOv2, a self - supervised pre - trained foundation model, on radiological images and whether its feature representation is general enough to be applied to radiological image analysis. Therefore, this paper explores these issues by comprehensively evaluating the performance of DINOv2 on radiological images, especially in disease classification and organ segmentation tasks. ### Research Background and Motivation 1. **Importance of Medical Images**: Radiological images are an indispensable tool in modern medicine for accurate and timely diagnosis. 2. **Existing Challenges**: - **High Cost of Labeled Data**: High - quality labeled data requires the time and effort of professional radiologists and is costly. - **Limited Generalization Ability**: Existing models perform well within a specific range but poorly in different domains, tasks, or with slight changes in data distribution. 3. **Advantages of Foundation Models**: Foundation models reduce the dependence on labeled data and improve the generalization ability and robustness of the model by pre - training on large - scale datasets. ### Research Methods 1. **Dataset**: This paper uses nine publicly available radiological benchmark datasets, covering X - ray, CT, and MRI examinations, involving disease classification and organ segmentation tasks. 2. **Evaluation Settings**: - **kNN**: Nearest - neighbor classification based on the normalized features of the last backbone layer. - **Linear Probing**: Attach a linear layer on top of the backbone layer. - **Few - shot Learning**: Train a linear layer on the frozen features. - **Fine - tuning**: Including parameter - efficient fine - tuning methods (such as LoRA and BitFit) and end - to - end fine - tuning. 3. **Model Comparison**: This paper compares DINOv2 with other supervised, weakly - supervised, and self - supervised models, including CLIP, OpenCLIP, MAE, and MSN, etc. ### Main Contributions 1. **Comprehensive Evaluation of DINOv2**: This paper comprehensively evaluates the performance of DINOv2 in multiple radiological modalities and different evaluation settings, especially in disease classification and organ segmentation tasks. 2. **Analysis of Cross - task Generalization Ability**: Analyzes the performance of DINOv2 relative to other models in cross - task generalization ability, and finds that DINOv2 outperforms self - supervised and weakly - supervised models in classification tasks and also performs well in segmentation tasks. 3. **Parameter - efficient Fine - tuning**: Studies the effect of parameter - efficient fine - tuning using the DINOv2 ViT - g/14 model, and finds that on large - scale X - ray classification datasets, the performance of PEFT can be comparable to that of end - to - end fine - tuning, but using less than 1% of the number of parameters. ### Conclusion This paper proves the superior performance and cross - task generalization ability of DINOv2 in disease classification and organ segmentation tasks by comprehensively evaluating the performance of DINOv2 on radiological images. These results provide valuable insights for optimizing medical image pre - training strategies and enhancing the understanding of the bridging role of DINOv2 between natural image and radiological image analysis.

Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks

Comparative Analysis of ImageNet Pre-Trained Deep Learning Models and DINOv2 in Medical Imaging Classification

Assessing the Performance of the DINOv2 Self-supervised Learning Vision Transformer Model for the Segmentation of the Left Atrium from MRI Images

Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?

MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging

Block Expanded DINORET: Adapting Natural Domain Foundation Models for Retinal Imaging Without Catastrophic Forgetting

Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning

RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision

Deep neural models for automated multi-task diagnostic scan management—quality enhancement, view classification and report generation

DinoBloom: A Foundation Model for Generalizable Cell Embeddings in Hematology

Foundation models in gastrointestinal endoscopic AI: Impact of architecture, pre-training approach and data efficiency

Foundation Models in Radiology: What, How, When, Why and Why Not

Medical Slice Transformer: Improved Diagnosis and Explainability on 3D Medical Images with DINOv2

DINOv2 Rocks Geological Image Analysis: Classification, Segmentation, and Interpretability

Are Natural Domain Foundation Models Useful for Medical Image Classification?

Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data

Expert-level vision-language foundation model for real-world radiology and comprehensive evaluation

Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains

Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision

Computer Vision Foundation Models in Endoscopy: Proof of Concept in Oropharyngeal Cancer

Exploring deep learning radiomics for classifying osteoporotic vertebral fractures in X-ray images