Abstract:Video object segmentation (VOS) has made significant progress with matching-based methods, but most approaches still show two problems. Firstly, they apply a complicated and redundant two-extractor pipeline to use more reference frames for cues, increasing the models’ parameters and complexity. Secondly, most of these methods neglect the spatial relationships (inside each frame) and do not fully model the temporal relationships (among different frames), i.e., they need adequate modeling of spatial-temporal relationships. In this paper, to address the two problems, we propose a unified transformer-based framework for VOS, a compact and unified single-extractor pipeline with strong spatial and temporal interaction ability. Specifically, to slim the common-used two-extractor pipeline while keeping the model’s effectiveness and flexibility, we design a single dynamic feature extractor with an ingenious dynamic input adapter to encode two significant inputs, i.e., reference sets (historical frames with predicted masks) and query frame (current frame), respectively. Moreover, the relationships among different frames and inside every frame are crucial for this task. We introduce a vision transformer to exploit and model both the temporal and spatial relationships simultaneously. By the cascaded design of the proposed dynamic feature extractor, transformer-based relationship module, and target-enhanced segmentation, our model implements a unified and compact pipeline for VOS. Extensive experiments demonstrate the superiority of our model over state-of-the-art methods on both DAVIS and YouTube-VOS datasets. We also explore potential solutions, such as sequence organizers, to improve the model’s efficiency. On DAVIS17 validation, we achieve ∼ 50 J & F ) drop in segmentation quality. Codes are available at https://github.com/sallymmx/TransVOS.git .

HierGAT: hierarchical spatial-temporal network with graph and transformer for video HOI detection

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

TransVOS: Video Object Segmentation with Transformers

Human-Object Interaction Prediction in Videos through Gaze Following

Spatial Parsing and Dynamic Temporal Pooling Networks for Human-Object Interaction Detection

Spatial-temporal Graph Transformer Network for Skeleton-Based Temporal Action Segmentation

HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization

Video Instance Segmentation Using Graph Matching Transformer

Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection

Exploring Spatio–Temporal Graph Convolution for Video-Based Human–Object Interaction Recognition

A Hierarchical Spatio-Temporal Model for Human Activity Recognition.

TAM-VT: Transformation-Aware Multi-scale Video Transformer for Segmentation and Tracking

Toward a Unified Transformer-Based Framework for Scene Graph Generation and Human-Object Interaction Detection

Real-time Human-Centric Segmentation for Complex Video Scenes

Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detection

Geometric Features Enhanced Human-Object Interaction Detection

Hierarchical Spatiotemporal Transformers for Video Object Segmentation

TMHOI: Translational Model for Human-Object Interaction Detection

Spatial-temporal hypergraph based on dual-stage attention network for multi-view data lightweight action recognition

Hierarchical Graph Pattern Understanding for Zero-Shot VOS

Hierarchical Graph Pattern Understanding for Zero-Shot Video Object Segmentation