SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree

Shuangrui Ding,Rui Qian,Xiaoyi Dong,Pan Zhang,Yuhang Zang,Yuhang Cao,Yuwei Guo,Dahua Lin,Jiaqi Wang

2024-10-22

Abstract:The Segment Anything Model 2 (SAM 2) has emerged as a powerful foundation model for object segmentation in both images and videos, paving the way for various downstream video applications. The crucial design of SAM 2 for video segmentation is its memory module, which prompts object-aware memories from previous frames for current frame prediction. However, its greedy-selection memory design suffers from the "error accumulation" problem, where an errored or missed mask will cascade and influence the segmentation of the subsequent frames, which limits the performance of SAM 2 toward complex long-term videos. To this end, we introduce SAM2Long, an improved training-free video object segmentation strategy, which considers the segmentation uncertainty within each frame and chooses the video-level optimal results from multiple segmentation pathways in a constrained tree search manner. In practice, we maintain a fixed number of segmentation pathways throughout the video. For each frame, multiple masks are proposed based on the existing pathways, creating various candidate branches. We then select the same fixed number of branches with higher cumulative scores as the new pathways for the next frame. After processing the final frame, the pathway with the highest cumulative score is chosen as the final segmentation result. Benefiting from its heuristic search design, SAM2Long is robust toward occlusions and object reappearances, and can effectively segment and track objects for complex long-term videos. Notably, SAM2Long achieves an average improvement of 3.0 points across all 24 head-to-head comparisons, with gains of up to 5.3 points in J&F on long-term video object segmentation benchmarks such as SA-V and LVOS. The code is released at <a class="link-external link-https" href="https://github.com/Mark12Ding/SAM2Long" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in the video object segmentation task, the existing Segment Anything Model 2 (SAM 2) has the problem of "error accumulation" when processing long - time videos. That is, once there is an error or omission in the segmentation of a certain frame, this error will gradually accumulate and affect the segmentation results of subsequent frames, especially in complex scenarios with frequent occlusions and object reappearances. This limits the performance of SAM 2 when dealing with complex long - term videos.

To overcome this problem, the authors propose SAM2Long, an improved video object segmentation strategy without training. SAM2Long effectively alleviates the "error accumulation" problem by considering the segmentation uncertainty in each frame and selecting the optimal video - level result from multiple segmentation paths in a constrained tree - search manner. Specifically, SAM2Long maintains a fixed number of segmentation paths at each time step. For each frame, it generates multiple candidate branches based on the existing paths, then selects a fixed number of branches with higher cumulative scores as new paths, and finally selects the path with the highest cumulative score as the final segmentation result. This method not only improves the model's robustness to occlusions and object reappearances but also significantly and consistently outperforms the performance of SAM 2 in five VOS benchmark tests.

### Key Technical Points
1. **Constrained Tree Memory Structure**:
- **Multi - path Exploration**: At each time step, SAM2Long maintains multiple possible correct mask paths and explores multiple segmentation hypotheses through a tree - like structure.
- **Path Selection and Pruning**: After generating multiple candidate branches at each time step, select a fixed number of branches with the highest cumulative scores to control the growth of the tree and reduce computational and memory overhead.

2. **Uncertainty Handling**:
- **High - score Mask Selection**: When the maximum absolute occlusion score exceeds the threshold, select the high - score mask.
- **Diversity Maintenance**: When all paths are uncertain, select masks with different shapes to avoid incorrect convergence.

3. **Object - aware Memory Bank Construction**:
- **Memory Frame Selection**: Select frames containing high - quality segmentation masks according to the predicted occlusion scores and IoU scores, and filter out frames with poor occlusion or segmentation quality.
- **Memory Attention Modulation**: Use the occlusion scores to modulate the contribution of memory entries and emphasize more reliable memory entries.

### Experimental Results
- **Performance Improvement**: SAM2Long significantly outperforms SAM 2 in multiple VOS benchmark tests, especially in processing long - time videos and scenarios with frequent occlusions.
- **Robustness Enhancement**: Through the constrained tree memory structure and uncertainty handling, SAM2Long can better handle occlusions and object reappearances.
- **Generalization Ability**: In unseen categories, SAM2Long shows stronger generalization ability.

In conclusion, SAM2Long significantly improves the performance and robustness of SAM 2 in processing complex long - term videos by introducing the constrained tree memory structure and uncertainty handling mechanism.