Integrating Spatial Prior Adapter for Enhancing SAM Performance in Medical Image Segmentation

Jihong Hu,Yinhao Li,Lanfen Lin,Yen-Wei Chen

DOI: https://doi.org/10.1109/GCCE59613.2023.10315265

2023-01-01

Abstract:The Segment Anything Model (SAM) has emerged as a focal point of interest within the domain of image segmentation. Capitalizing on extensive pretraining with vast datasets and an interactive prompting mechanism, SAM stands out as a formidable base model for natural image segmentation tasks. However, a number of empirical studies indicate that SAM's performance diminishes substantially when applied to standard medical segmentation tasks. This is largely attributed to the modal and feature disparities between natural and medical images. In response to this challenge, this study introduces a novel, parameter-efficient fine-tuning approach termed the Spatial Prior Adapter (SPA). SPA leverages convolutional neural networks (CNNs) to capture multi-scale features, which are seamlessly amalgamated with the features harnessed from the SAM image encoder. Notably, during the model's fine-tuning phase, the SAM image encoder is frozen, enabling exclusive refinement of the SPA for optimal parameter calibration. Experimental evaluations on two medical segmentation datasets validate the effectiveness of our proposed method.