Abstract:Segment Anything Model (SAM) has demonstrated impressive performance on a wide range of natural image segmentation tasks. However, its performance significantly deteriorates when directly applied to medical domain, due to the remarkable differences between natural images and medical images. Some researchers have attempted to train SAM on large scale medical datasets. However, poor zero-shot performance is observed from the experimental results. In this context, inspired by the superior performance of U-Net-like models in medical image segmentation, we propose SAMUNet, a new foundation model which incorporates U-Net to the original SAM, to fully leverage the powerful contextual modeling ability of convolutions. To be specific, we parallel a convolutional branch in the image encoder, which is trained independently with the vision Transformer branch frozen. Additionally, we employ multi-scale fusion in the mask decoder, to facilitate accurate segmentation of objects with different scales. We train SAM-UNet on SA-Med2D-16M, the largest 2-dimensional medical image segmentation dataset to date, yielding a universal pretrained model for medical images. Extensive experiments are conducted to evaluate the performance of the model, and state-of-the-art result is achieved, with a dice similarity coefficient score of 0.883 on SA-Med2D-16M dataset. Specifically, in zero-shot segmentation experiments, our model not only significantly outperforms previous large medical SAM models across all modalities, but also substantially mitigates the performance degradation seen on unseen modalities. It should be highlighted that SAM-UNet is an efficient and extensible foundation model, which can be further fine-tuned for other downstream tasks in medical community. The code is available at <a class="link-external link-https" href="https://github.com/Hhankyangg/sam-unet" rel="external noopener nofollow">this https URL</a>.

Accuracy of Segment-Anything Model (SAM) in medical image segmentation tasks

Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets

Segment Anything Model for Medical Images?

Segment Anything Model for Medical Image Analysis: an Experimental Study

SAM Fails to Segment Anything? – SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More

How Segment Anything Model (SAM) Boost Medical Image Segmentation?

Evaluating segment anything model (SAM) on MRI scans of brain tumors

Segment Anything in Medical Images

Foundation Models for Biomedical Image Segmentation: A Survey

Towards Segment Anything Model (SAM) for Medical Image Segmentation: A Survey

Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging

Segment anything model for medical image segmentation: Current applications and future directions

Empirical Evaluation of the Segment Anything Model (SAM) for Brain Tumor Segmentation

SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model

No More Training: SAM's Zero-Shot Transfer Capabilities for Cost-Efficient Medical Image Segmentation

Segment anything model 2: an application to 2D and 3D medical images

Zero-shot performance of the Segment Anything Model (SAM) in 2D medical imaging: A comprehensive evaluation and practical guidelines

SAM-UNet:Enhancing Zero-Shot Segmentation of SAM for Universal Medical Images

Exploiting the Segment Anything Model (SAM) for Lung Segmentation in Chest X-ray Images

SimSAM: Zero-shot Medical Image Segmentation via Simulated Interaction