Foundation Models for Biomedical Image Segmentation: A Survey

Ho Hin Lee,Yu Gu,Theodore Zhao,Yanbo Xu,Jianwei Yang,Naoto Usuyama,Cliff Wong,Mu Wei,Bennett A. Landman,Yuankai Huo,Alberto Santamaria-Pang,Hoifung Poon

2024-01-15

Abstract:Recent advancements in biomedical image analysis have been significantly driven by the Segment Anything Model (SAM). This transformative technology, originally developed for general-purpose computer vision, has found rapid application in medical image processing. Within the last year, marked by over 100 publications, SAM has demonstrated its prowess in zero-shot learning adaptations for medical imaging. The fundamental premise of SAM lies in its capability to segment or identify objects in images without prior knowledge of the object type or imaging modality. This approach aligns well with tasks achievable by the human visual system, though its application in non-biological vision contexts remains more theoretically challenging. A notable feature of SAM is its ability to adjust segmentation according to a specified resolution scale or area of interest, akin to semantic priming. This adaptability has spurred a wave of creativity and innovation in applying SAM to medical imaging. Our review focuses on the period from April 1, 2023, to September 30, 2023, a critical first six months post-initial publication. We examine the adaptations and integrations of SAM necessary to address longstanding clinical challenges, particularly in the context of 33 open datasets covered in our analysis. While SAM approaches or achieves state-of-the-art performance in numerous applications, it falls short in certain areas, such as segmentation of the carotid artery, adrenal glands, optic nerve, and mandible bone. Our survey delves into the innovative techniques where SAM's foundational approach excels and explores the core concepts in translating and applying these models effectively in diverse medical imaging scenarios.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily explores the applications and challenges of the Segment Anything Model (SAM) in the field of biomedical image segmentation. SAM was initially developed as a model for general computer vision tasks, capable of segmenting objects in images without prior knowledge of the target type or imaging modality. This review paper focuses on the rapid development and application of SAM in the field of biomedical image analysis since its release (from April 1, 2023, to September 30, 2023).

The main issues mentioned in the paper include:
1. **Performance of SAM in zero-shot learning scenarios**: Evaluating SAM's ability to be directly applied to medical image processing without any specific medical domain training.
2. **Adaptation for specific medical domains**: Discussing how to optimize SAM's performance to meet the needs of medical image segmentation through various methods. This includes strategies such as prompt tuning, adapter tuning, and full model tuning.
3. **Extension to 3D imaging**: Exploring how to extend the SAM model, originally designed for 2D images, to 3D medical images to better handle 3D data generated by CT and MRI.
4. **Knowledge distillation and weak supervision**: Introducing how to use SAM for weakly supervised or semi-supervised learning to reduce dependence on high-quality annotated data and improve the model's performance on limited datasets.

By conducting an in-depth analysis of these aspects, the paper aims to reveal the advantages and limitations of SAM technology in the field of biomedical image segmentation and propose directions for future research. Additionally, the paper specifically points out the shortcomings of SAM in segmenting certain complex anatomical regions (such as the carotid artery, adrenal gland, optic nerve, and mandible), providing directions for further research.