Abstract:Video object segmentation (VOS) has made significant progress with matching-based methods, but most approaches still show two problems. Firstly, they apply a complicated and redundant two-extractor pipeline to use more reference frames for cues, increasing the models’ parameters and complexity. Secondly, most of these methods neglect the spatial relationships (inside each frame) and do not fully model the temporal relationships (among different frames), i.e., they need adequate modeling of spatial-temporal relationships. In this paper, to address the two problems, we propose a unified transformer-based framework for VOS, a compact and unified single-extractor pipeline with strong spatial and temporal interaction ability. Specifically, to slim the common-used two-extractor pipeline while keeping the model’s effectiveness and flexibility, we design a single dynamic feature extractor with an ingenious dynamic input adapter to encode two significant inputs, i.e., reference sets (historical frames with predicted masks) and query frame (current frame), respectively. Moreover, the relationships among different frames and inside every frame are crucial for this task. We introduce a vision transformer to exploit and model both the temporal and spatial relationships simultaneously. By the cascaded design of the proposed dynamic feature extractor, transformer-based relationship module, and target-enhanced segmentation, our model implements a unified and compact pipeline for VOS. Extensive experiments demonstrate the superiority of our model over state-of-the-art methods on both DAVIS and YouTube-VOS datasets. We also explore potential solutions, such as sequence organizers, to improve the model’s efficiency. On DAVIS17 validation, we achieve ∼ 50 J & F ) drop in segmentation quality. Codes are available at https://github.com/sallymmx/TransVOS.git .

Context-aware Deformable Alignment for Video Object Segmentation

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

Multilevel Spatial-Temporal Feature Aggregation for Video Object Detection

Video Object Segmentation with Dynamic Memory Networks and Adaptive Object Alignment.

Learning Position and Target Consistency for Memory-based Video Object Segmentation

DMVOS: Discriminative Matching for Real-time Video Object Segmentation

Complementary Coarse-to-Fine Matching for Video Object Segmentation

Tsanet: Temporal and Scale Alignment for Unsupervised Video Object Segmentation

Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement

Boosting Video Object Segmentation Via Space-time Correspondence Learning

Beyond Appearance: Multi-Frame Spatio-Temporal Context Memory Networks for Efficient and Robust Video Object Segmentation

COMatchNet: Co-Attention Matching Network for Video Object Segmentation.

DeVOS: Flow-Guided Deformable Transformer for Video Object Segmentation

Kernel based local matching network for video object segmentation

Adaptive Selection of Reference Frames for Video Object Segmentation.

Look Before You Match: Instance Understanding Matters in Video Object Segmentation

Adaptive Multi-scale Iterative Optimized Video Object Segmentation Based on Correlation Enhancement

Adaptive Online Learning For Video Object Segmentation

Object-based spatial similarity for semi-supervised video object segmentation

Towards Robust Video Object Segmentation with Adaptive Object Calibration

Target Aware Adaptive Tracking for Unsupervised Video Object Segmentation