Abstract:Video object segmentation (VOS) has made significant progress with matching-based methods, but most approaches still show two problems. Firstly, they apply a complicated and redundant two-extractor pipeline to use more reference frames for cues, increasing the models’ parameters and complexity. Secondly, most of these methods neglect the spatial relationships (inside each frame) and do not fully model the temporal relationships (among different frames), i.e., they need adequate modeling of spatial-temporal relationships. In this paper, to address the two problems, we propose a unified transformer-based framework for VOS, a compact and unified single-extractor pipeline with strong spatial and temporal interaction ability. Specifically, to slim the common-used two-extractor pipeline while keeping the model’s effectiveness and flexibility, we design a single dynamic feature extractor with an ingenious dynamic input adapter to encode two significant inputs, i.e., reference sets (historical frames with predicted masks) and query frame (current frame), respectively. Moreover, the relationships among different frames and inside every frame are crucial for this task. We introduce a vision transformer to exploit and model both the temporal and spatial relationships simultaneously. By the cascaded design of the proposed dynamic feature extractor, transformer-based relationship module, and target-enhanced segmentation, our model implements a unified and compact pipeline for VOS. Extensive experiments demonstrate the superiority of our model over state-of-the-art methods on both DAVIS and YouTube-VOS datasets. We also explore potential solutions, such as sequence organizers, to improve the model’s efficiency. On DAVIS17 validation, we achieve ∼ 50 J & F ) drop in segmentation quality. Codes are available at https://github.com/sallymmx/TransVOS.git .

Lightweight Video Object Segmentation Based on ConvGRU.

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

Fast Real-Time Video Object Segmentation with a Tangled Memory Network

Learning Quality-aware Dynamic Memory for Video Object Segmentation

Efficient video object segmentation based on Gaussian mixture model and Markov random field

Efficient Video Object Segmentation by Graph-Cut

Video object segmentation via couple streams and feature memory

Multi-Granularity Context Network for Efficient Video Semantic Segmentation

Semi-supervised Video Object Segmentation with Recurrent Neural Network

Learning Spatial-Semantic Features for Robust Video Object Segmentation

Video object segmentation by Multi-Scale Pyramidal Multi-Dimensional LSTM with generated depth context

Dual Temporal Memory Network for Efficient Video Object Segmentation

Beyond Appearance: Multi-Frame Spatio-Temporal Context Memory Networks for Efficient and Robust Video Object Segmentation

Design Light-weight 3D Convolutional Networks for Video Recognition Temporal Residual, Fully Separable Block, and Fast Algorithm

Learning Video Object Segmentation with Visual Memory

Spatial-information Guided Adaptive Context-aware Network for Efficient RGB-D Semantic Segmentation

Video Object Segmentation via Structural Feature Reconfiguration

Video Object Segmentation by Learning Location-Sensitive Embeddings

Video object segmentation by motion-based sequential feature clustering.

A Lightweight YOLOv5-Based Model with Feature Fusion and Dilation Convolution for Image Segmentation

Automatic Video Object Segmentation Algorithm for Multiple Scenes