Abstract:Segmentation in medical imaging is a critical component for the diagnosis, monitoring, and treatment of various diseases and medical conditions. Presently, the medical segmentation landscape is dominated by numerous specialized deep learning models, each fine-tuned for specific segmentation tasks and image modalities. The recently-introduced Segment Anything Model (SAM) employs the ViT neural architecture and harnesses a massive training dataset to segment nearly any object; however, its suitability to the medical domain has not yet been investigated. In this study, we explore the zero-shot performance of SAM in medical imaging by implementing eight distinct prompt strategies across six datasets from four imaging modalities, including X-ray, ultrasound, dermatoscopy, and colonoscopy. Our findings reveal that SAM's zero-shot performance is not only comparable to, but in certain cases, surpasses the current state-of-the-art. Based on these results, we propose practical guidelines that require minimal interaction while consistently yielding robust outcomes across all assessed contexts. The source code, along with a demonstration of the recommended guidelines, can be accessed at <a class="link-external link-https" href="https://github.com/Malta-Lab/SAM-zero-shot-in-Medical-Imaging" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate the zero - shot performance of the Segment Anything Model (SAM) in 2D medical images and provide practical guidelines. Specifically, the researchers hope to understand whether SAM can effectively segment medical images under different imaging modalities, such as X - ray, ultrasound, dermoscopy, and colonoscopy images, without being trained on specific medical image datasets. In addition, they also explore the impact of different prompting strategies on SAM performance and propose practical guidelines that doctors can adopt to achieve robust segmentation results with minimal interaction. ### Main research questions: 1. **Zero - shot performance evaluation**: Can SAM effectively segment medical images without being trained on a specific medical image dataset? 2. **Performance in different imaging modalities**: How does SAM perform in different imaging modalities such as X - ray, ultrasound, dermoscopy, and colonoscopy? 3. **Impact of prompting strategies**: What is the impact of different prompting strategies (such as center point, random point, distributed random point, bounding box and its variants) on SAM's segmentation performance? 4. **Practical guidelines**: Based on the experimental results, propose practical guidelines for doctors on how to use SAM for medical image segmentation in practical applications. ### Research methods: - **Datasets**: Six datasets from four imaging modalities were used, including ISIC 2018, HAM10000, Montgomery - Shenzhen, X - ray Images of Hip Joints, CVC - ClinicDB, and Breast Ultrasound Images. - **Prompting strategies**: Eight different prompting strategies were designed, including center point (CP), random point (RP), distributed random point (RP3 and RP5), bounding box (BB) and its variants (BBS5, BBS10, and BBS20). - **Evaluation metrics**: The Dice Similarity Coefficient (DSC) was used as the main metric for evaluating segmentation performance. ### Main findings: - **Bounding box prompting strategy**: The bounding box (BB) and its variants (BBS5 and BBS10) showed the best performance in all datasets and could maintain high - quality segmentation even with slight inaccuracies. - **Multi - point prompting strategy**: Increasing the number of input points can improve the model performance, but still cannot surpass the bounding box prompting strategy. - **Impact of model size**: The ViT - B model has performance comparable to the larger ViT - L and ViT - H models, and even performs better in some cases, and its GPU memory requirements are lower, making it suitable for more cost - effective hardware. - **Comparison with existing methods**: On some datasets, SAM's zero - shot performance exceeds the existing state - of - the - art methods, especially on datasets with a small amount of data. ### Practical guidelines: - **Initial prompt**: It is recommended to use the bounding box prompt to delimit the target area and select the most appropriate segmentation result among the three generated predictions. - **Refined segmentation**: If necessary, the segmentation result can be further refined by adding additional point prompts to exclude unwanted areas or include missed areas. Through these studies, the authors not only verified the potential of SAM in medical image segmentation but also provided specific guidance and suggestions for practical applications.

Zero-shot performance of the Segment Anything Model (SAM) in 2D medical imaging: A comprehensive evaluation and practical guidelines

SAM3D: Zero-Shot Semi-Automatic Segmentation in 3D Medical Images with the Segment Anything Model

SimSAM: Zero-shot Medical Image Segmentation via Simulated Interaction

SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model

No More Training: SAM's Zero-Shot Transfer Capabilities for Cost-Efficient Medical Image Segmentation

Segment Anything Model (SAM) for Medical Image Segmentation: A Preliminary Review

Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation

Segment Anything Model for Medical Image Analysis: an Experimental Study

Segment anything model 2: an application to 2D and 3D medical images

Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging

Accuracy of Segment-Anything Model (SAM) in medical image segmentation tasks

Interactive 3D Medical Image Segmentation with SAM 2

Medical SAM 2: Segment medical images as video via Segment Anything Model 2

Increasing SAM Zero-Shot Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation

Segment Anything in Medical Images and Videos: Benchmark and Deployment

S-SAM: SVD-based Fine-Tuning of Segment Anything Model for Medical Image Segmentation

A Short Review and Evaluation of SAM2's Performance in 3D CT Image Segmentation

Zero-shot 3D Segmentation of Abdominal Organs in CT Scans Using Segment Anything Model 2

Unleashing the Potential of SAM2 for Biomedical Images and Videos: A Survey

MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation

Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets