Abstract:Video object segmentation (VOS) has made significant progress with matching-based methods, but most approaches still show two problems. Firstly, they apply a complicated and redundant two-extractor pipeline to use more reference frames for cues, increasing the models’ parameters and complexity. Secondly, most of these methods neglect the spatial relationships (inside each frame) and do not fully model the temporal relationships (among different frames), i.e., they need adequate modeling of spatial-temporal relationships. In this paper, to address the two problems, we propose a unified transformer-based framework for VOS, a compact and unified single-extractor pipeline with strong spatial and temporal interaction ability. Specifically, to slim the common-used two-extractor pipeline while keeping the model’s effectiveness and flexibility, we design a single dynamic feature extractor with an ingenious dynamic input adapter to encode two significant inputs, i.e., reference sets (historical frames with predicted masks) and query frame (current frame), respectively. Moreover, the relationships among different frames and inside every frame are crucial for this task. We introduce a vision transformer to exploit and model both the temporal and spatial relationships simultaneously. By the cascaded design of the proposed dynamic feature extractor, transformer-based relationship module, and target-enhanced segmentation, our model implements a unified and compact pipeline for VOS. Extensive experiments demonstrate the superiority of our model over state-of-the-art methods on both DAVIS and YouTube-VOS datasets. We also explore potential solutions, such as sequence organizers, to improve the model’s efficiency. On DAVIS17 validation, we achieve ∼ 50 J & F ) drop in segmentation quality. Codes are available at https://github.com/sallymmx/TransVOS.git .

Visual Features Extraction Through Spatiotemporal Slice Analysis

Retrieval Method For Video Content In Different Format Based On Spatiotemporal Features

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

Video Identification Using Spatio-temporal Salient Points

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

STC: Spatio-Temporal Contrastive Learning for Video Instance Segmentation.

Spatio-Temporal Segmentation with Depth-Inferred Videos of Static Scenes

Adaptive Slicing Method of the Spatiotemporal Event Stream Obtained from a Dynamic Vision Sensor

Format-Independent Motion Content Description Based on Spatiotemporal Visual Sensitivity

Incremetal Spatio-Temporal Feature Extraction and Retrieval for Large Video Database

A Video Classification Method Based on Spatiotemporal Detail Attention and Feature Fusion

Spatiotemporal Video Copy Detection Based on Visual Perception Analyses

Spatiotemporal Feature Extraction Based on Invariance Representation

Spatio-temporal feature learning for enhancing video quality based on screen content characteristics

Spatiotemporal interest point detector exploiting appearance and motion-variation information

Spatial-Temporal Salient Unit Detection Based on Features in Video Sequences

Dual Spatial-Temporal Feature Pyramid With Decoupled Temporal Mining for Video Semantic Segmentation

Transformer Based Sptial-Temporal Extraction Model for Video Anomaly Detection

Spatiotemporal Feature Learning for Event-Based Vision

Object Extraction by Spatio-Temporal Assembling

Spatiotemporal segmentation for compact video representation