Abstract:Most deep learning approaches to comprehensive semantic modeling of 3D indoor spaces require costly dense annotations in the 3D domain. In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations. The key idea of our approach is to design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images, fusing cross-domain features into volumetric embeddings to predict complete 3D geometry, color, and semantics with only 2D labeling which can be either manual or machine-generated. Our key technical innovation is to leverage differentiable rendering of color and semantics to bridge 2D observations and unknown 3D space, using the observed RGB images and 2D semantics as supervision, respectively. We additionally develop a learning pipeline and corresponding method to enable learning from imperfect predicted 2D labels, which could be additionally acquired by synthesizing in an augmented set of virtual training views complementing the original real captures, enabling more efficient self-supervision loop for semantics. As a result, our end-to-end trainable solution jointly addresses geometry completion, colorization, and semantic mapping from limited RGB-D images, without relying on any 3D ground-truth information. Our method achieves the state-of-the-art performance of semantic scene completion on two large-scale benchmark datasets MatterPort3D and ScanNet, surpasses baselines even with costly 3D annotations in predicting both geometry and semantics. To our knowledge, our method is also the first 2D-driven method addressing completion and semantic segmentation of real-world 3D scans simultaneously.

Zero-Shot Multi-Object Scene Completion

Up-to-Down Network: Fusing Multi-Scale Context for 3D Semantic Scene Completion

In-Hand 3D Object Reconstruction from a Monocular RGB Video

Zero-Shot 3d Pose Estimation of Unseen Object by Two-Step Rgb-D Fusion

Instance-Aware Monocular 3D Semantic Scene Completion

Behind the Veil: Enhanced Indoor 3D Scene Reconstruction with Occluded Surfaces Completion

To Complete or to Estimate, That is the Question: A Multi-Task Approach to Depth Completion and Monocular Depth Estimation

Diorama: Unleashing Zero-shot Single-view 3D Scene Modeling

SceneComplete: Open-World 3D Scene Completion in Complex Real World Environments for Robot Manipulation

O$^2$-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model

SSR-2D: Semantic 3D Scene Reconstruction from 2D Images

CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation

Voxel- and Bird's-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

Deep Learned Full-3D Object Completion from Single View

Geometry-semantic Aware for Monocular 3D Semantic Scene Completion

Three Cars Approaching within 100m! Enhancing Distant Geometry by Tri-Axis Voxel Scanning for Camera-based Semantic Scene Completion

Semantic Scene Completion using Local Deep Implicit Functions on LiDAR Data

ET-Former: Efficient Triplane Deformable Attention for 3D Semantic Scene Completion From Monocular Camera

DepthSSC: Depth-Spatial Alignment and Dynamic Voxel Resolution for Monocular 3D Semantic Scene Completion

ESC-Net: Alleviating Triple Sparsity on 3D LiDAR Point Clouds for Extreme Sparse Scene Completion

ZeroPose: CAD-Prompted Zero-shot Object 6D Pose Estimation in Cluttered Scenes