Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation

Saiyang Na,Yuzhi Guo,Feng Jiang,Hehuan Ma,Junzhou Huang

2024-01-24

Abstract:In the rapidly evolving field of AI research, foundational models like BERT and GPT have significantly advanced language and vision tasks. The advent of pretrain-prompting models such as ChatGPT and Segmentation Anything Model (SAM) has further revolutionized image segmentation. However, their applications in specialized areas, particularly in nuclei segmentation within medical imaging, reveal a key challenge: the generation of high-quality, informative prompts is as crucial as applying state-of-the-art (SOTA) fine-tuning techniques on foundation models. To address this, we introduce Segment Any Cell (SAC), an innovative framework that enhances SAM specifically for nuclei segmentation. SAC integrates a Low-Rank Adaptation (LoRA) within the attention layer of the Transformer to improve the fine-tuning process, outperforming existing SOTA methods. It also introduces an innovative auto-prompt generator that produces effective prompts to guide segmentation, a critical factor in handling the complexities of nuclei segmentation in biomedical imaging. Our extensive experiments demonstrate the superiority of SAC in nuclei segmentation tasks, proving its effectiveness as a tool for pathologists and researchers. Our contributions include a novel prompt generation strategy, automated adaptability for diverse segmentation tasks, the innovative application of Low-Rank Attention Adaptation in SAM, and a versatile framework for semantic segmentation challenges.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address key challenges in nuclear segmentation, particularly when applying the Segment Anything Model (SAM) in the field of medical imaging. Specifically:

1. **Importance of High-Quality Prompts**: The paper highlights that generating high-quality, information-rich prompts is as crucial as applying state-of-the-art fine-tuning techniques in nuclear segmentation tasks. Low-quality prompts lead to inaccurate segmentation results.

2. **Limitations of SAM in Medical Image Segmentation**: Although SAM performs well in general image segmentation tasks, it has limitations in medical image segmentation (especially nuclear segmentation) mainly due to insufficient training data and lack of domain-specific knowledge.

3. **Need for Automated Prompt Generation**: Manually annotating each nucleus in a large number of nuclear images is impractical. Therefore, developing methods that can automatically and efficiently generate high-quality prompts becomes essential.

### Main Contributions of the Paper

1. **Novel Prompt Generation Strategy**: A new prompt generation method is proposed, which significantly enhances SAM's performance in nuclear segmentation tasks through high-quality prompts. This method can be used complementarily with existing state-of-the-art fine-tuning methods.

2. **Automatic Adaptability**: The proposed Segment Any Cell (SAC) model is fully automated and can easily adapt to various nuclear segmentation tasks, simplifying the workflow for pathologists and researchers.

3. **Low-Rank Attention Adaptation Innovation**: Low-Rank Attention Adaptation is introduced, which improves the accuracy and efficiency of SAM in nuclear segmentation without increasing the number of parameters.

4. **General Framework**: A general fine-tuning framework based on SAM is proposed, applicable to different semantic segmentation tasks, addressing similar challenges faced in nuclear segmentation.