All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning

Can Cui,Ruining Deng,Quan Liu,Tianyuan Yao,Shunxing Bao,Lucas W. Remedios,Yucheng Tang,Yuankai Huo

2023-08-29

Abstract:The Segment Anything Model (SAM) is a recently proposed prompt-based segmentation model in a generic zero-shot segmentation approach. With the zero-shot segmentation capacity, SAM achieved impressive flexibility and precision on various segmentation tasks. However, the current pipeline requires manual prompts during the inference stage, which is still resource intensive for biomedical image segmentation. In this paper, instead of using prompts during the inference stage, we introduce a pipeline that utilizes the SAM, called all-in-SAM, through the entire AI development workflow (from annotation generation to model finetuning) without requiring manual prompts during the inference stage. Specifically, SAM is first employed to generate pixel-level annotations from weak prompts (e.g., points, bounding box). Then, the pixel-level annotations are used to finetune the SAM segmentation model rather than training from scratch. Our experimental results reveal two key findings: 1) the proposed pipeline surpasses the state-of-the-art (SOTA) methods in a nuclei segmentation task on the public Monuseg dataset, and 2) the utilization of weak and few annotations for SAM finetuning achieves competitive performance compared to using strong pixel-wise annotated data.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the problem of reducing the cost and workload of manual annotation in biomedical image segmentation tasks, particularly in the task of nuclear segmentation. Specifically, the existing Segment Anything Model (SAM) requires manual prompts (such as bounding boxes or points) for nuclear segmentation, which is still very time-consuming in practical applications. Therefore, the paper proposes a new method—All-in-SAM, which generates high-quality pixel-level annotations using weakly labeled data and fine-tunes the pre-trained SAM model based on this, thereby achieving efficient nuclear segmentation without manual prompts during the inference phase.

The main objectives include:
1. Generating high-quality pixel-level annotations using weakly labeled data (such as bounding boxes) to reduce annotation costs.
2. Developing a label-efficient fine-tuning pipeline that enables the SAM model to achieve performance comparable to existing state-of-the-art methods with a small amount of labeled data.

Through these methods, the paper aims to improve the application efficiency of SAM in nuclear segmentation tasks, reduce the workload of manual annotation, and maintain high-precision segmentation results.