Abstract:Pixelwise single object tracking is challenging due to the competition of running speeds and segmentation accuracy. Current state-of-the-art real-time approaches seamlessly connect tracking and segmentation by sharing computation of the backbone network, e.g., SiamMask and D3S fork a light branch from the tracking model to predict segmentation mask. Although efficient, directly reusing features from tracking networks may harm the segmentation accuracy, since background clutter in the backbone feature tends to introduce false positives in segmentation. To mitigate this problem, we propose a unified tracking-retrieval-segmentation framework consisting of an attention retrieval network (ARN) and an iterative feedback network (IFN). Instead of segmenting the target inside the bounding box, the proposed framework performs soft spatial constraints on backbone features to obtain an accurate global segmentation map. Concretely, in ARN, a look-up-table (LUT) is first built by sufficiently using the information of the first frame. By retrieving it, a target-aware attention map is generated to suppress the negative influence of background clutter. To ulteriorly refine the contour of the segmentation, IFN iteratively enhances the features at different resolutions by taking the predicted mask as feedback guidance. Our framework sets a new state of the art on the recent pixelwise tracking benchmark VOT2020 and runs at 40 fps. Notably, the proposed model surpasses SiamMask by 11.7/4.2/5.5 points on VOT2020, DAVIS2016, and DAVIS2017, respectively. Code is available at https://github.com/JudasDie/SOTS.

Fine-Grained Human-Centric Tracklet Segmentation with Single Frame Supervision

Exploit Spatiotemporal Contextual Information for 3D Single Object Tracking Via Memory Networks

Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework With Spatio-Temporal Collaboration

STC: Spatio-Temporal Contrastive Learning for Video Instance Segmentation.

C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentation

Toward High Quality Multi-Object Tracking and Segmentation Without Mask Supervision

Video Segmentation by Tracking Many Figure-Ground Segments

Segment-Tube: Spatio-Temporal Action Localization in Untrimmed Videos with Per-Frame Segmentation.

Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation

Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation

Exploring Frame Segmentation Networks for Temporal Action Localization

Tracking Objects and Activities with Attention for Temporal Sentence Grounding

Coarse-to-Fine Video Instance Segmentation With Factorized Conditional Appearance Flows

TempNet: Online Semantic Segmentation on Large-scale Point Cloud Series

SG-TCN: Semantic Guidance Temporal Convolutional Network for Action Segmentation.

Video Object Segmentation via Structural Feature Reconfiguration

Learning Local and Global Temporal Contexts for Video Semantic Segmentation

Temporal Segment Transformer for Action Segmentation

Toward Accurate Pixelwise Object Tracking via Attention Retrieval

Streaming Video Temporal Action Segmentation In Real Time