Novel adaptation of video segmentation to 3D MRI: efficient zero-shot knee segmentation with SAM2

Andrew Seohwan Yu,Mohsen Hariri,Xuecen Zhang,Mingrui Yang,Vipin Chaudhary,Xiaojuan Li

2024-08-09

Abstract:Intelligent medical image segmentation methods are rapidly evolving and being increasingly applied, yet they face the challenge of domain transfer, where algorithm performance degrades due to different data distributions between source and target domains. To address this, we introduce a method for zero-shot, single-prompt segmentation of 3D knee MRI by adapting Segment Anything Model 2 (SAM2), a general-purpose segmentation model designed to accept prompts and retain memory across frames of a video. By treating slices from 3D medical volumes as individual video frames, we leverage SAM2's advanced capabilities to generate motion- and spatially-aware predictions. We demonstrate that SAM2 can efficiently perform segmentation tasks in a zero-shot manner with no additional training or fine-tuning, accurately delineating structures in knee MRI scans using only a single prompt. Our experiments on the Osteoarthritis Initiative Zuse Institute Berlin (OAI-ZIB) dataset reveal that SAM2 achieves high accuracy on 3D knee bone segmentation, with a testing Dice similarity coefficient of 0.9643 on tibia. We also present results generated using different SAM2 model sizes, different prompt schemes, as well as comparative results from the SAM1 model deployed on the same dataset. This breakthrough has the potential to revolutionize medical image analysis by providing a scalable, cost-effective solution for automated segmentation, paving the way for broader clinical applications and streamlined workflows.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main purpose of this paper is to address several key issues in medical image segmentation and propose an efficient method for zero-shot segmentation of three-dimensional (3D) MRI images. Specifically:

1. **Problems with traditional segmentation models**: Traditional medical image segmentation models typically require a large amount of annotated data for training and are tailored to specific datasets or disease types, limiting their generalizability across different medical imaging tasks.
2. **Introduction of SAM1**: To address the above issues, the researchers proposed the Segment Anything Model 1 (SAM1), a promptable image segmentation method that achieves precise and context-sensitive segmentation through user interactive inputs (such as points and bounding boxes), thereby enhancing its generalizability and flexibility.
3. **Challenges in video segmentation**: With the growth of multimedia content, much visual data exists in video form, necessitating a universal visual segmentation system that can handle both images and videos.
4. **Extension to SAM2**: Based on this need, the Segment Anything Model 2 (SAM2) extends the capabilities of SAM1 to not only handle images but also process video data. SAM2 introduces the Promptable Visual Segmentation (PVS) task, allowing users to define regions of interest on any video frame and iteratively update these regions in subsequent frames.
5. **Application to 3D MRI**: This paper further applies SAM2 to 3D MRI images by treating slices in the 3D volume as individual video frames, leveraging SAM2's video segmentation capabilities to achieve effective segmentation of three-dimensional medical images.

In summary, the paper demonstrates how the SAM2 model can be successfully applied to zero-shot segmentation of 3D MRI images, particularly for knee MRI images, improving segmentation accuracy and efficiency, and simplifying the workflow of medical image analysis.