Abstract:We address the problem of semi-supervised video object segmentation (VOS), where the masks of objects of interests are given in the first frame of an input video. To deal with challenging cases where objects are occluded or missing, previous work relies on greedy data association strategies that make decisions for each frame individually. In this paper, we propose a novel approach to defer the decision making for a target object in each frame, until a global view can be established with the entire video being taken into consideration. Our approach is in the same spirit as Multiple Hypotheses Tracking (MHT) methods, making several critical adaptations for the VOS problem. We employ the bounding box (bbox) hypothesis for tracking tree formation, and the multiple hypotheses are spawned by propagating the preceding bbox into the detected bbox proposals within a gated region starting from the initial object mask in the first frame. The gated region is determined by a gating scheme which takes into account a more comprehensive motion model rather than the simple Kalman filtering model in traditional MHT. To further design more customized algorithms tailored for VOS, we develop a novel mask propagation score instead of the appearance similarity score that could be brittle due to large deformations. The mask propagation score, together with the motion score, determines the affinity between the hypotheses during tree pruning. Finally, a novel mask merging strategy is employed to handle mask conflicts between objects. Extensive experiments on challenging datasets demonstrate the effectiveness of the proposed method, especially in the case of object missing.

Video Object Co-Segmentation via Subspace Clustering and Quadratic Pseudo-Boolean Optimization in an MRF Framework

An algorithm for automatic segmentation of video objects based on intra-frame image partition

Automatic Extraction Of Moving Objects In Video Sequences Based On Spatio-Temporal Information

Automatic Extraction of Video Object Plane by Integrating Spatio-Temporal Information

Efficient video object segmentation based on Gaussian mixture model and Markov random field

Robust Video Object Cosegmentation.

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

An automatic segmentation algorithm for moving objects in video sequences under multi-constraints

Multiresolution Segmentation of Video Objects in the Compression Domain

A Novel Video Object Tracking Approach Based on Kernel Density Estimation and Markov Random Field

Object Segmentation Using Graph Cuts for the H.264 Compressed Video with Moving Background

Real-time spatiotemporal segmentation of video objects in the H.264 compressed domain

Non-rigid Video Object Segmentation Based on Semantic Multi-level Framework.

Video Object Segmentation with Dynamic Query Modulation

MHP-VOS: Multiple Hypotheses Propagation for Video Object Segmentation

Monocular Video Foreground/Background Segmentation by Tracking Spatial-Color Gaussian Mixture Models

Co-Segmentation Of Image Pairs With Quadratic Global Constraint In Mrfs

Video Object Segmentation by Hierarchical Localized Classification of Regions

Automatic Object Cosegmentation In Sparse Multiview Images

An Effective Method for Foreground Segmentation of Video