Abstract:Most deep learning approaches to comprehensive semantic modeling of 3D indoor spaces require costly dense annotations in the 3D domain. In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations. The key idea of our approach is to design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images, fusing cross-domain features into volumetric embeddings to predict complete 3D geometry, color, and semantics with only 2D labeling which can be either manual or machine-generated. Our key technical innovation is to leverage differentiable rendering of color and semantics to bridge 2D observations and unknown 3D space, using the observed RGB images and 2D semantics as supervision, respectively. We additionally develop a learning pipeline and corresponding method to enable learning from imperfect predicted 2D labels, which could be additionally acquired by synthesizing in an augmented set of virtual training views complementing the original real captures, enabling more efficient self-supervision loop for semantics. As a result, our end-to-end trainable solution jointly addresses geometry completion, colorization, and semantic mapping from limited RGB-D images, without relying on any 3D ground-truth information. Our method achieves the state-of-the-art performance of semantic scene completion on two large-scale benchmark datasets MatterPort3D and ScanNet, surpasses baselines even with costly 3D annotations in predicting both geometry and semantics. To our knowledge, our method is also the first 2D-driven method addressing completion and semantic segmentation of real-world 3D scans simultaneously.

3D Model Semantic Automatic Annotation Based on X3D Scene

A Novel Semantic Annotation Algorithm for Models Based on Associated Scene.

Learning 3 D Scene Synthesis from Annotated RGB-D Images

An Overview of Semantics Processing in Content-Based 3D Model Retrieval

Domain-Specific Modeling and Semantic Alignment for Image-Based 3d Model Retrieval

Automatic Semantic Modeling of Indoor Scenes from Low-Quality RGB-D Data Using Contextual Information

A Semantic Context Model For Automatic Image Annotation

High-level semantic video annotation based on 3D scene structure analysis

Three-Dimensional Structure Measurement And Optimization Method Of Indoor Scene Based On Single Image

Method of generating three-dimensional scene model

3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera

3D Models Semantic Retrieval Method in Combination with Descriptive Text

SSR-2D: Semantic 3D Scene Reconstruction from 2D Images

An Interactive Approach to Semantic Modeling of Indoor Scenes with an RGBD Camera

Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models

Semi-automatic Video Content Annotation

Automatic 3D Indoor Scene Modeling from Single Panorama

Open-set Hierarchical Semantic Segmentation for 3D Scene

Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding

Singe Image-Based Data-Driven Indoor Scenes Modeling

3D Scene Graph Generation from Point Clouds