Abstract:Video Object Segmentation (VOS) has emerged as an increasingly important problem with availability of larger datasets and more complex and realistic settings, which involve long videos with global motion (e.g, in egocentric settings), depicting small objects undergoing both rigid and non-rigid (including state) deformations. While a number of recent approaches have been explored for this task, these data characteristics still present challenges. In this work we propose a novel, clip-based DETR-style encoder-decoder architecture, which focuses on systematically analyzing and addressing aforementioned challenges. Specifically, we propose a novel transformation-aware loss that focuses learning on portions of the video where an object undergoes significant deformations -- a form of "soft" hard examples mining. Further, we propose a multiplicative time-coded memory, beyond vanilla additive positional encoding, which helps propagate context across long videos. Finally, we incorporate these in our proposed holistic multi-scale video transformer for tracking via multi-scale memory matching and decoding to ensure sensitivity and accuracy for long videos and small objects. Our model enables on-line inference with long videos in a windowed fashion, by breaking the video into clips and propagating context among them. We illustrate that short clip length and longer memory with learned time-coding are important design choices for improved performance. Collectively, these technical contributions enable our model to achieve new state-of-the-art (SoTA) performance on two complex egocentric datasets -- VISOR and VOST, while achieving comparable to SoTA results on the conventional VOS benchmark, DAVIS'17. A series of detailed ablations validate our design choices as well as provide insights into the importance of parameter choices and their impact on performance.

Multiview Transformers for Video Recognition

TransVOS: Video Object Segmentation with Transformers

MMViT: Multiscale Multiview Vision Transformers

Hierarchical Separable Video Transformer for Snapshot Compressive Imaging

Multi-view Convolutional Vision Transformer for 3D Object Recognition.

OVPT: Optimal Viewset Pooling Transformer for 3D Object Recognition.

Disentangling 3D/4D Facial Affect Recognition with Faster Multi-View Transformer

TRecViT: A Recurrent Video Transformer

MgMViT: Multi-Granularity and Multi-Scale Vision Transformer for Efficient Action Recognition

Multi-view 3D Reconstruction from Video with Transformer.

Feature‐enhanced representation with transformers for multi‐view stereo

Multi-View Stereo with Transformer

Multi-entity Video Transformers for Fine-Grained Video Representation Learning

Imvs: Integrating Multi-View Information on Multiple Scales for 3D Object Recognition

Multiview Detection with Shadow Transformer (and View-Coherent Data Augmentation)

Temporally Efficient Vision Transformer for Video Instance Segmentation

TAM-VT: Transformation-Aware Multi-scale Video Transformer for Segmentation and Tracking

A Video Is Worth Three Views: Trigeminal Transformers for Video-Based Person Re-Identification

MED-VT++: Unifying Multimodal Learning with a Multiscale Encoder-Decoder Video Transformer

DVANet: Disentangling View and Action Features for Multi-View Action Recognition