Abstract:Video object segmentation (VOS) has made significant progress with matching-based methods, but most approaches still show two problems. Firstly, they apply a complicated and redundant two-extractor pipeline to use more reference frames for cues, increasing the models’ parameters and complexity. Secondly, most of these methods neglect the spatial relationships (inside each frame) and do not fully model the temporal relationships (among different frames), i.e., they need adequate modeling of spatial-temporal relationships. In this paper, to address the two problems, we propose a unified transformer-based framework for VOS, a compact and unified single-extractor pipeline with strong spatial and temporal interaction ability. Specifically, to slim the common-used two-extractor pipeline while keeping the model’s effectiveness and flexibility, we design a single dynamic feature extractor with an ingenious dynamic input adapter to encode two significant inputs, i.e., reference sets (historical frames with predicted masks) and query frame (current frame), respectively. Moreover, the relationships among different frames and inside every frame are crucial for this task. We introduce a vision transformer to exploit and model both the temporal and spatial relationships simultaneously. By the cascaded design of the proposed dynamic feature extractor, transformer-based relationship module, and target-enhanced segmentation, our model implements a unified and compact pipeline for VOS. Extensive experiments demonstrate the superiority of our model over state-of-the-art methods on both DAVIS and YouTube-VOS datasets. We also explore potential solutions, such as sequence organizers, to improve the model’s efficiency. On DAVIS17 validation, we achieve ∼ 50 J & F ) drop in segmentation quality. Codes are available at https://github.com/sallymmx/TransVOS.git .

Novel tile segmentation scheme for omnidirectional video

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

Asymmetric representation for 3D panoramic video

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

An Optimal Spatial-Temporal Smoothness Approach for Tile-Based 360-Degree Video Streaming

Texture-Aware Spherical Rotation for High Efficiency Omnidirectional Intra Video Coding

Optimal Volumetric Video Streaming with Hybrid Saliency based Tiling

Tile-Based Rate Assignment for 360-Degree Video Based on Spatio-Temporal Activity Metrics.

Selection of the Most Efficient Tile Size in Tile-Based Cylinder Panoramic Video Coding and Transmission

Transcoing-based Data Transmission of Sphere Panoramic Videos

Content Adaptive Tiling Method Based On User Access Preference For Streaming Panoramic Video

The Most Efficient Tile Size in Tile-Based Cylinder Panoramic Video Coding and Its Selection under Restriction of Bandwidth

RGB-D Video Segmentation via Geodesic Spatio-Temporal Propagation

Optimized Video Coding For Omnidirectional Videos

An Adaptive Frame Skipping and Vop Interpolation Algorithm for Video Object Segmentation

Spatio-Temporal Segmentation with Depth-Inferred Videos of Static Scenes

Line-preserving video stitching for asymmetric cameras

Mosaic Based View Enlargement For Moving Objects In Moving Pictures

Polar Square Projection for Panoramic Video

An Efficient Spherical Video Sampling Scheme Based on Cube Model

Adaptive Transmission Strategy for Non-Uniform Coding of 360∘ Videos