Abstract:We propose an end-to-end learning framework for segmenting generic objects in both images and videos. Given a novel image or video, our approach produces a pixel-level mask for all "object-like" regions-even for object categories never seen during training. We formulate the task as a structured prediction problem of assigning an object/background label to each pixel, implemented using a deep fully convolutional network. When applied to a video, our model further incorporates a motion stream, and the network learns to combine both appearance and motion and attempts to extract all prominent objects whether they are moving or not. Beyond the core model, a second contribution of our approach is how it leverages varying strengths of training annotations. Pixel-level annotations are quite difficult to obtain, yet crucial for training a deep network approach for segmentation. Thus we propose ways to exploit weakly labeled data for learning dense foreground segmentation. For images, we show the value in mixing object category examples with image-level labels together with relatively few images with boundary-level annotations. For video, we show how to bootstrap weakly annotated videos together with the network trained for image segmentation. Through experiments on multiple challenging image and video segmentation benchmarks, our method offers consistently strong results and improves the state-of-the-art for fully automatic segmentation of generic (unseen) objects. In addition, we demonstrate how our approach benefits image retrieval and image retargeting, both of which flourish when given our high-quality foreground maps. Code, models, and videos are at: http://vision.cs.utexas.edu/projects/pixelobjectness/.

Key-segments for video object segmentation

An algorithm for automatic segmentation of video objects based on intra-frame image partition

Automatic foreground extraction in video

Object Segmentation Using Graph Cuts for the H.264 Compressed Video with Moving Background

Pixel Objectness: Learning to Segment Generic Objects Automatically in Images and Videos

Object Segmentation From Long Video Sequences

Video Object Segmentation by Learning Location-Sensitive Embeddings

Robust Video Object Cosegmentation.

Motion Objects Segmentation Using a New Level Set Based Method

Non-rigid Video Object Segmentation Based on Semantic Multi-level Framework.

A novel video abstraction method based on fast clustering of the regions of interest in key frames

Reliable and Dynamic Appearance Modeling and Label Consistency Enforcing for Fast and Coherent Video Object Segmentation With the Bilateral Grid

Hierarchical video object segmentation

Efficient video object segmentation based on Gaussian mixture model and Markov random field

An Effective Method for Foreground Segmentation of Video

Automatic Segmentation of Objects of Interest in Video: a Unified Framework

Video Object Co-Segmentation via Subspace Clustering and Quadratic Pseudo-Boolean Optimization in an MRF Framework

Video object segmentation by tracking regions

Video Segmentation by Tracking Many Figure-Ground Segments

Asymmetric Label Propagation for Video Object Segmentation.

Learning to Segment Moving Objects in Videos