Abstract:Video object segmentation (VOS) has made significant progress with matching-based methods, but most approaches still show two problems. Firstly, they apply a complicated and redundant two-extractor pipeline to use more reference frames for cues, increasing the models’ parameters and complexity. Secondly, most of these methods neglect the spatial relationships (inside each frame) and do not fully model the temporal relationships (among different frames), i.e., they need adequate modeling of spatial-temporal relationships. In this paper, to address the two problems, we propose a unified transformer-based framework for VOS, a compact and unified single-extractor pipeline with strong spatial and temporal interaction ability. Specifically, to slim the common-used two-extractor pipeline while keeping the model’s effectiveness and flexibility, we design a single dynamic feature extractor with an ingenious dynamic input adapter to encode two significant inputs, i.e., reference sets (historical frames with predicted masks) and query frame (current frame), respectively. Moreover, the relationships among different frames and inside every frame are crucial for this task. We introduce a vision transformer to exploit and model both the temporal and spatial relationships simultaneously. By the cascaded design of the proposed dynamic feature extractor, transformer-based relationship module, and target-enhanced segmentation, our model implements a unified and compact pipeline for VOS. Extensive experiments demonstrate the superiority of our model over state-of-the-art methods on both DAVIS and YouTube-VOS datasets. We also explore potential solutions, such as sequence organizers, to improve the model’s efficiency. On DAVIS17 validation, we achieve ∼ 50 J & F ) drop in segmentation quality. Codes are available at https://github.com/sallymmx/TransVOS.git .

End-to-End Video Scene Graph Generation with Temporal Propagation Transformer

Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation

TransVOS: Video Object Segmentation with Transformers

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

A Novel End-to-End Transformer for Scene Graph Generation

Temporally Consistent Dynamic Scene Graphs: An End-to-End Approach for Action Tracklet Generation

Dynamic Scene Graph Generation Via Temporal Prior Inference

SGTR+: End-to-end Scene Graph Generation with Transformer

DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation

Iterative Scene Graph Generation with Generative Transformers

Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation

Target Adaptive Context Aggregation for Video Scene Graph Generation

Toward a Unified Transformer-Based Framework for Scene Graph Generation and Human-Object Interaction Detection

Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection

Reasoning in Different Directions: Triplet Learning for Scene Graph Generation

Transformer-based Image Generation from Scene Graphs

OED: Towards One-stage End-to-End Dynamic Scene Graph Generation

GTR: A Grafting-Then-Reassembling Framework for Dynamic Scene Graph Generation

RelTR: Relation Transformer for Scene Graph Generation

Temporally Consistent Transformers for Video Generation

Video Semantic Segmentation Via Sparse Temporal Transformer.