Abstract:Conventional video matting outputs one alpha matte for all instances appearing in a video frame so that individual instances are not distinguished. While video instance segmentation provides time-consistent instance masks, results are unsatisfactory for matting applications, especially due to applied binarization. To remedy this deficiency, we propose Video Instance Matting~(VIM), that is, estimating alpha mattes of each instance at each frame of a video sequence. To tackle this challenging problem, we present MSG-VIM, a Mask Sequence Guided Video Instance Matting neural network, as a novel baseline model for VIM. MSG-VIM leverages a mixture of mask augmentations to make predictions robust to inaccurate and inconsistent mask guidance. It incorporates temporal mask and temporal feature guidance to improve the temporal consistency of alpha matte predictions. Furthermore, we build a new benchmark for VIM, called VIM50, which comprises 50 video clips with multiple human instances as foreground objects. To evaluate performances on the VIM task, we introduce a suitable metric called Video Instance-aware Matting Quality~(VIMQ). Our proposed model MSG-VIM sets a strong baseline on the VIM50 benchmark and outperforms existing methods by a large margin. The project is open-sourced at <a class="link-external link-https" href="https://github.com/SHI-Labs/VIM" rel="external noopener nofollow">this https URL</a>.

Temporally consistent video matting based on bilayer segmentation

Attention-guided Temporally Coherent Video Object Matting

Interactive space-time video matting approach

Video Editing with Temporal, Spatial and Appearance Consistency

Adaptive Background Matting Using Background Matching

Easy Matting - A Stroke Based Approach for Continuous Image Matting

A Progressive Tri-level Segmentation Approach for Topology-Change-Aware Video Matting.

Spatio-Temporal Segmentation with Depth-Inferred Videos of Static Scenes

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

Deep Video Matting via Spatio-Temporal Alignment and Aggregation

Automatic Matting Based on Fusion of Blur Estimation and Saliency Detection

Multi-guided-based image matting via boundary detection

Contour-assistance-based video matting localization

Robust High-Resolution Video Matting with Temporal Guidance

Weakly Supervised Image Matting Via Patch Clustering

Video Instance Matting

Coarse Semantic Guided Alpha Matting Via Simultaneous Foreground and Background Estimation

Efficient video object segmentation based on Gaussian mixture model and Markov random field

A Late Fusion CNN for Digital Matting

Live Video Montage with a Rotating Camera

dugMatting: Decomposed-Uncertainty-Guided Matting