Abstract:We focus on the problem of semantic segmentation based on RGB-D data, with emphasis on analyzing cluttered indoor scenes containing many instances from many visual categories. Our approach is based on a parametric figure-ground intensity and depth-constrained proposal process that generates spatial layout hypotheses at multiple locations and scales in the image followed by a sequential inference algorithm that integrates the proposals into a complete scene estimate. Our contributions can be summarized as proposing the following: (1) a generalization of parametric max flow figure-ground proposal methodology to take advantage of intensity and depth information, in order to systematically and efficiently generate the breakpoints of an underlying spatial model in polynomial time, (2) new region description methods based on second-order pooling over multiple features constructed using both intensity and depth channels, (3) an inference procedure that can resolve conflicts in overlapping spatial partitions, and handles scenes with a large number of objects category instances, of very different scales, (4) extensive evaluation of the impact of depth, as well as the effectiveness of a large number of descriptors, both pre-designed and automatically obtained using deep learning, in a difficult RGB-D semantic segmentation problem with 92 classes. We report state of the art results in the challenging NYU Depth v2 dataset, extended for RMRC 2013 Indoor Segmentation Challenge, where currently the proposed model ranks first, with an average score of 24.61% and a number of 39 classes won. Moreover, we show that by combining second-order and deep learning features, over 15% relative accuracy improvements can be additionally achieved. In a scene classification benchmark, our methodology further improves the state of the art by 24%.

Object Proposal via Depth Connectivity Constrained Grouping.

Object Proposal on RGB-D Images Via Elastic Edge Boxes

Depth-Aware Layered Edge for Object Proposal

Elastic Edge Boxes for Object Proposal on RGB-D Images.

3D-SSD: Learning Hierarchical Features from RGB-D Images for Amodal 3D Object Detection

Locating 3D Object Proposals: A Depth-Based Online Approach

Interactive Hierarchical Object Proposals

Object Segmentation Ensuring Consistency Across Multi-Viewpoint Images

Object Pose Estimation Using Color Images and Predicted Depth Maps

Constrained Parametric Proposals and Pooling Methods for Semantic Segmentation in RGB-D Images

Depth Cue Enhancement and Guidance Network for RGB-D Salient Object Detection

Texture Complexity Based Redundant Regions Ranking for Object Proposal.

Context-Aware Video Object Proposals.

RGB-D Video Segmentation via Geodesic Spatio-Temporal Propagation

Weakly-Supervised RGBD Video Object Segmentation

Adaptive Integration Of Depth And Color For Objectness Estimation

Deep Contour and Symmetry Scored Object Proposal.

Adaptive video object proposals by a context-aware model

Object Detection and Tracking under Occlusion for Object-Level RGB-D Video Segmentation

Object Level Depth Reconstruction for Category Level 6D Object Pose Estimation from Monocular RGB Image

BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth