Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning

Haofeng Liu,Erli Zhang,Junde Wu,Mingxuan Hong,Yueming Jin
2024-08-15
Abstract:Surgical video segmentation is a critical task in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, the Segment Anything Model 2 (SAM2) framework has shown superior advancements in image and video segmentation. However, SAM2 struggles with efficiency due to the high computational demands of processing high-resolution images and complex and long-range temporal dynamics in surgical videos. To address these challenges, we introduce Surgical SAM 2 (SurgSAM-2), an advanced model to utilize SAM2 with an Efficient Frame Pruning (EFP) mechanism, to facilitate real-time surgical video segmentation. The EFP mechanism dynamically manages the memory bank by selectively retaining only the most informative frames, reducing memory usage and computational cost while maintaining high segmentation accuracy. Our extensive experiments demonstrate that SurgSAM-2 significantly improves both efficiency and segmentation accuracy compared to the vanilla SAM2. Remarkably, SurgSAM-2 achieves a 3$\times$ FPS compared with SAM2, while also delivering state-of-the-art performance after fine-tuning with lower-resolution data. These advancements establish SurgSAM-2 as a leading model for surgical video analysis, making real-time surgical video segmentation in resource-constrained environments a feasible reality.
Computer Vision and Pattern Recognition,Artificial Intelligence,Robotics,Image and Video Processing
What problem does this paper attempt to address?
This paper attempts to solve the problem of how to achieve real - time and efficient surgical video segmentation in computer - assisted surgery. Specifically, the paper proposes improvement schemes for the following problems: 1. **High computational requirements**: Although the existing Segment Anything Model 2 (SAM2) model performs well in image and video segmentation, when dealing with high - resolution images and complex long - term dynamic changes (such as the long - term spatial dynamics in surgical videos), the computational requirements are too high, resulting in low efficiency. 2. **Redundant information**: In surgical videos, the semantic information between adjacent frames is highly similar, especially when the camera is fixed or moves slowly, resulting in the processing of a large amount of duplicate data, which further increases the computational burden. 3. **Memory management problems**: SAM2 adopts a mechanism of storing frames in sequence, which will lead to a large amount of redundant information retained in the memory, thus affecting the performance of real - time applications. To address these problems, the paper introduces Surgical SAM 2 (SurgSAM - 2), a model specifically optimized for surgical video segmentation. SurgSAM - 2 dynamically manages the memory bank by introducing the Efficient Frame Pruning (EFP) mechanism, selectively retains the most informative frames, reduces memory usage and computational cost, while maintaining high segmentation accuracy. ### Main contributions 1. **Proposing SurgSAM - 2**: An innovative model based on SAM2, specifically optimized for surgical video segmentation. 2. **EFP mechanism**: Through a cosine - similarity - based scoring system, it intelligently selects and prunes unimportant frames, thereby significantly improving computational efficiency and memory utilization. 3. **Experimental verification**: Extensive experiments were carried out on two benchmark datasets, EndoVis17 and EndoVis18, demonstrating the superiority of SurgSAM - 2 in terms of accuracy and efficiency. ### Experimental results The experimental results show that SurgSAM - 2 has a significant improvement compared to the original SAM2: - **FPS (Frames Per Second)**: It has increased by approximately 13.8%. - **Memory usage**: It has decreased by approximately 8.5%. - **Segmentation accuracy**: Under different settings (Full Mask, One Point, Five Points), SurgSAM - 2 shows higher J&F scores and Dice coefficients. These improvements enable SurgSAM - 2 to achieve efficient real - time surgical video segmentation in resource - constrained environments, providing strong support for clinical applications.