Abstract:Video object segmentation (VOS) has made significant progress with matching-based methods, but most approaches still show two problems. Firstly, they apply a complicated and redundant two-extractor pipeline to use more reference frames for cues, increasing the models’ parameters and complexity. Secondly, most of these methods neglect the spatial relationships (inside each frame) and do not fully model the temporal relationships (among different frames), i.e., they need adequate modeling of spatial-temporal relationships. In this paper, to address the two problems, we propose a unified transformer-based framework for VOS, a compact and unified single-extractor pipeline with strong spatial and temporal interaction ability. Specifically, to slim the common-used two-extractor pipeline while keeping the model’s effectiveness and flexibility, we design a single dynamic feature extractor with an ingenious dynamic input adapter to encode two significant inputs, i.e., reference sets (historical frames with predicted masks) and query frame (current frame), respectively. Moreover, the relationships among different frames and inside every frame are crucial for this task. We introduce a vision transformer to exploit and model both the temporal and spatial relationships simultaneously. By the cascaded design of the proposed dynamic feature extractor, transformer-based relationship module, and target-enhanced segmentation, our model implements a unified and compact pipeline for VOS. Extensive experiments demonstrate the superiority of our model over state-of-the-art methods on both DAVIS and YouTube-VOS datasets. We also explore potential solutions, such as sequence organizers, to improve the model’s efficiency. On DAVIS17 validation, we achieve ∼ 50 J & F ) drop in segmentation quality. Codes are available at https://github.com/sallymmx/TransVOS.git .

CSMB-VSS: video scene segmentation with cosine similarity matrix

Global and Compact Video Context Embedding for Video Semantic Segmentation

Contrasting Multi-Modal Similarity Framework for Video Scene Segmentation

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

Contour Based Automatic Scene Segmentation in Image Sequences

Temporal Scene Montage for Self-Supervised Video Scene Boundary Detection

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

Scene Segmentation Based on Video Structure and Spectral Methods

OS-MSL: One Stage Multimodal Sequential Link Framework for Scene Segmentation and Classification

Video scene segmentation and semantic representation using a novel scheme

A New Approach For Video Scene Boundary Detection

Optimized Video Scene Segmentation.

Scene Consistency Representation Learning for Video Scene Segmentation

Learning Spatial-Semantic Features for Robust Video Object Segmentation

Research on the Method of Video Semantic Scene Constructing Based on SVM

S3-Net: A Fast and Lightweight Video Scene Understanding Network by Single-shot Segmentation.

Video Shot Boundary Detection and Scene Segmentation

TDSNet: A Temporal Difference Based Network for Video Semantic Segmentation

Object-based spatial similarity for semi-supervised video object segmentation

Learning a Contextual Multi-Thread Model for Movie/TV Scene Segmentation

Video Object Co-Segmentation from Noisy Videos by a Multi-Level Hypergraph Model.