Abstract:Video instance segmentation (VIS) is a critical task with diverse applications, including autonomous driving and video editing. Existing methods often underperform on complex and long videos in real world, primarily due to two factors. Firstly, offline methods are limited by the tightly-coupled modeling paradigm, which treats all frames equally and disregards the interdependencies between adjacent frames. Consequently, this leads to the introduction of excessive noise during long-term temporal alignment. Secondly, online methods suffer from inadequate utilization of temporal information. To tackle these challenges, we propose a decoupling strategy for VIS by dividing it into three independent sub-tasks: segmentation, tracking, and refinement. The efficacy of the decoupling strategy relies on two crucial elements: 1) attaining precise long-term alignment outcomes via frame-by-frame association during tracking, and 2) the effective utilization of temporal information predicated on the aforementioned accurate alignment outcomes during refinement. We introduce a novel referring tracker and temporal refiner to construct the \textbf{D}ecoupled \textbf{VIS} framework (\textbf{DVIS}). DVIS achieves new SOTA performance in both VIS and VPS, surpassing the current SOTA methods by 7.3 AP and 9.6 VPQ on the OVIS and VIPSeg datasets, which are the most challenging and realistic benchmarks. Moreover, thanks to the decoupling strategy, the referring tracker and temporal refiner are super light-weight (only 1.69\% of the segmenter FLOPs), allowing for efficient training and inference on a single GPU with 11G memory. The code is available at \href{<a class="link-external link-https" href="https://github.com/zhang-tao-whu/DVIS" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/zhang-tao-whu/DVIS" rel="external noopener nofollow">this https URL</a>}.

HumVis: Human-Centric Visual Analysis System

Real-time Human-Centric Segmentation for Complex Video Scenes

A Unified Framework for Human-centric Point Cloud Video Understanding

Human-Centric Visual Analysis: Tasks and Progress

Human Centric Visual Analysis with Deep Learning

HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling

HUMA'20

UniHCP: A Unified Model for Human-Centric Perceptions

Generating Human-Centric Visual Cues for Human-Object Interaction Detection via Large Vision-Language Models

Human-to-Human Interaction Detection

HUMBI: A Large Multiview Dataset of Human Body Expressions and Benchmark Challenge

HUMA'21

Human Instance Segmentation and Tracking via Data Association and Single-stage Detector

FT-HID: a large-scale RGB-D dataset for first- and third-person human interaction analysis

Human-Centric Images and Videos Analysis

HUMA'21: 2nd International Workshop on Human-centric Multimedia Analysis

A Generic Framework for Human-Machine Hybrid Recognition Based on Wearable Visual Computing

Human Mesh Recovery from Arbitrary Multi-view Images

Collaborative Scalable Visual Compression for Human-Centered Videos.

Contributions of Shape, Texture, and Color in Visual Recognition

DVIS: Decoupled Video Instance Segmentation Framework