Learning Spatial-Semantic Features for Robust Video Object Segmentation

Xin Li,Deshui Miao,Zhenyu He,Yaowei Wang,Huchuan Lu,Ming-Hsuan Yang

2024-07-10

Abstract:Tracking and segmenting multiple similar objects with complex or separate parts in long-term videos is inherently challenging due to the ambiguity of target parts and identity confusion caused by occlusion, background clutter, and long-term variations. In this paper, we propose a robust video object segmentation framework equipped with spatial-semantic features and discriminative object queries to address the above issues. Specifically, we construct a spatial-semantic network comprising a semantic embedding block and spatial dependencies modeling block to associate the pretrained ViT features with global semantic features and local spatial features, providing a comprehensive target representation. In addition, we develop a masked cross-attention module to generate object queries that focus on the most discriminative parts of target objects during query propagation, alleviating noise accumulation and ensuring effective long-term query propagation. The experimental results show that the proposed method set a new state-of-the-art performance on multiple datasets, including the DAVIS2017 test (89.1%), YoutubeVOS 2019 (88.5%), MOSE (75.1%), LVOS test (73.0%), and LVOS val (75.1%), which demonstrate the effectiveness and generalization capacity of the proposed method. We will make all source code and trained models publicly available.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

This paper aims to address several key challenges in video object segmentation (VOS), especially when dealing with target objects with complex or separated parts. Specifically, the paper attempts to solve the following problems:

1. **Dramatic changes in target appearance**: In long - time videos, the target object may undergo significant appearance changes, which makes existing VOS methods difficult to maintain accurate tracking and segmentation.
2. **Occlusion and background clutter**: Occlusion and background clutter can lead to blurring and identity confusion of target parts, especially when dealing with multiple similar objects.
3. **Incomplete prediction of target parts**: Existing methods mainly rely on pixel - level correlations, which results in incomplete prediction masks when dealing with targets with complex structures or separated parts.
4. **Noise accumulation in query propagation**: Existing query - based methods often introduce noise and errors when updating target queries, leading to long - term tracking failures.

To address these problems, the authors propose a new VOS framework that combines spatial - semantic features and discriminative object queries. Specific contributions are as follows:

- **Spatial - semantic feature learning**: By constructing a spatial - semantic block containing a semantic embedding module and a spatial - dependency - modeling module, this framework can effectively utilize the global semantic information provided by the pre - trained ViT model and the local spatial information provided by the multi - scale CNN network to generate a comprehensive target representation.
- **Discriminative query mechanism**: A discriminative query - propagation module has been developed, which can pay more attention to the representative parts of the target object, thereby reducing noise accumulation during query propagation and improving robustness in long - term videos.
- **Extensive experimental verification**: Extensive experiments have been carried out on five benchmark datasets, including DAVIS 2017, YouTubeVOS 2018, YouTubeVOS 2019, LVOS and MOSE. The experimental results show that this method has achieved new state - of - the - art performance on all of these datasets.

Through these innovations, this paper provides a more robust and efficient video object segmentation solution, especially suitable for dealing with objects with complex structures and long - term deformations.