Cross Prompting Consistency with Segment Anything Model for Semi-supervised Medical Image Segmentation

Juzheng Miao,Cheng Chen,Keli Zhang,Jie Chuai,Quanzheng Li,Pheng-Ann Heng

2024-07-07

Abstract:Semi-supervised learning (SSL) has achieved notable progress in medical image segmentation. To achieve effective SSL, a model needs to be able to efficiently learn from limited labeled data and effectively exploiting knowledge from abundant unlabeled data. Recent developments in visual foundation models, such as the Segment Anything Model (SAM), have demonstrated remarkable adaptability with improved sample efficiency. To harness the power of foundation models for application in SSL, we propose a cross prompting consistency method with segment anything model (CPC-SAM) for semi-supervised medical image segmentation. Our method employs SAM's unique prompt design and innovates a cross-prompting strategy within a dual-branch framework to automatically generate prompts and supervisions across two decoder branches, enabling effectively learning from both scarce labeled and valuable unlabeled data. We further design a novel prompt consistency regularization, to reduce the prompt position sensitivity and to enhance the output invariance under different prompts. We validate our method on two medical image segmentation tasks. The extensive experiments with different labeled-data ratios and modalities demonstrate the superiority of our proposed method over the state-of-the-art SSL methods, with more than 9% Dice improvement on the breast cancer segmentation task.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use limited labeled data and a large amount of unlabeled data to improve the performance of the model in medical image segmentation tasks. Specifically, the paper proposes a semi - supervised learning framework based on Segment Anything Model (SAM) - the Cross - Prompt Consistency method (CPC - SAM), aiming to improve the learning efficiency of the model under the condition of a small amount of labeled data in the following ways:

1. **Quickly learn from a small amount of labeled data**: By fine - tuning the SAM model, it can quickly learn discriminative information from the limited labeled data.
2. **Effectively utilize a large amount of unlabeled data**: Utilizing the unique prompt mechanism of SAM, a cross - prompt strategy is designed to automatically generate prompts and supervision signals within a two - branch framework, thereby fully exploiting the knowledge in the unlabeled data.
3. **Reduce prompt position sensitivity**: A new Prompt Consistency Regularization (PCR) strategy is designed to reduce the sensitivity of SAM to different prompt positions and enhance the invariance of the output under different prompts.

The paper verifies the effectiveness of the proposed method through experiments on two medical image segmentation tasks. Especially when the labeled data is extremely limited, its performance is significantly better than existing semi - supervised learning methods. For example, in the breast cancer segmentation task, using only 10 labeled ultrasound images, the Dice coefficient of the CPC - SAM method is more than 9% higher than that of multiple strong baseline methods.