Spatio-Temporal Segmentation with Depth-Inferred Videos of Static Scenes

Hanqing Jiang,Guofeng Zhang,H. Bao

2010-01-01

Abstract:Extracting spatio-temporally consistent segments from a video sequence is a challenging problem due to the complexity of color, motion and occlusions. Most existing spatio-temporal segmentation approaches rely on pairwise motion estimation, which have inherent difficulties in handling large displacement with significant occlusions. This paper presents a novel spatio-temporal segmentation method for depth-inferred videos. The depth data of input videos can be estimated beforehand using a multiview stereo technique. Our framework consists of two steps. In the first step, in order to make the extracted 2D segments temporally consistent, we introduce a spatial segmentation based on the probabilistic boundary maps, by collecting the boundary statistics in a video. In the second step, the consistent 2D segments in different frames are matched to initialize the volume segments. Then we compute the segment probability for each pixel by projecting it to other frames to collect the statistics, and incorporate it into the spatio-temporal segmentation energy to explicitly enforce temporal coherence constraint. The spatio-temporal segmentation results are iteratively refined in a video, so that a set of spatio-temporally consistent volume segments are finally achieved. The effectiveness of our automatic method is demonstrated using a variety of challenging video examples.