Promise:Prompt-driven 3D Medical Image Segmentation Using Pretrained Image Foundation Models

Hao Li,Han Liu,Dewei Hu,Jiacheng Wang,Ipek Oguz

2023-11-14

Abstract:To address prevalent issues in medical imaging, such as data acquisition challenges and label availability, transfer learning from natural to medical image domains serves as a viable strategy to produce reliable segmentation results. However, several existing barriers between domains need to be broken down, including addressing contrast discrepancies, managing anatomical variability, and adapting 2D pretrained models for 3D segmentation tasks. In this paper, we propose ProMISe,a prompt-driven 3D medical image segmentation model using only a single point prompt to leverage knowledge from a pretrained 2D image foundation model. In particular, we use the pretrained vision transformer from the Segment Anything Model (SAM) and integrate lightweight adapters to extract depth-related (3D) spatial context without updating the pretrained weights. For robust results, a hybrid network with complementary encoders is designed, and a boundary-aware loss is proposed to achieve precise boundaries. We evaluate our model on two public datasets for colon and pancreas tumor segmentations, respectively. Compared to the state-of-the-art segmentation methods with and without prompt engineering, our proposed method achieves superior performance. The code is publicly available at <a class="link-external link-https" href="https://github.com/MedICL-VU/ProMISe" rel="external noopener nofollow">this https URL</a>.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address several key issues in medical image segmentation, including the difficulty of data acquisition, insufficient labels, and the challenges of transfer learning from natural images to the medical image domain. Specifically, directly using pre-trained 2D natural image models for 3D medical image segmentation often performs poorly for the following reasons:

1. **Contrast and Texture Differences**: Medical images have unique contrast and texture characteristics.
2. **Anatomical Differences**: Anatomical variations between individuals make medical image segmentation more challenging.
3. **Loss of Depth Information**: Slice-by-slice segmentation methods based on 2D pre-trained models ignore important depth-related spatial context in 3D medical data.

To address these issues, the paper proposes a new method called ProMISe, which achieves effective adaptation of pre-trained image foundation models through the following innovations:

- Using lightweight adapters to optimize cross-domain knowledge transfer and better capture fine-grained features.
- Designing a simple yet efficient boundary-aware loss function to improve segmentation accuracy in edge-blurred regions.
- Demonstrating its performance on two public datasets, proving that the method outperforms existing state-of-the-art segmentation methods in colon and pancreatic tumor segmentation tasks.

In summary, this study aims to improve the accuracy and robustness of 3D medical image segmentation through improved pre-trained model adaptation strategies and innovative loss function design.