Abstract:Most deep learning approaches to comprehensive semantic modeling of 3D indoor spaces require costly dense annotations in the 3D domain. In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations. The key idea of our approach is to design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images, fusing cross-domain features into volumetric embeddings to predict complete 3D geometry, color, and semantics with only 2D labeling which can be either manual or machine-generated. Our key technical innovation is to leverage differentiable rendering of color and semantics to bridge 2D observations and unknown 3D space, using the observed RGB images and 2D semantics as supervision, respectively. We additionally develop a learning pipeline and corresponding method to enable learning from imperfect predicted 2D labels, which could be additionally acquired by synthesizing in an augmented set of virtual training views complementing the original real captures, enabling more efficient self-supervision loop for semantics. As a result, our end-to-end trainable solution jointly addresses geometry completion, colorization, and semantic mapping from limited RGB-D images, without relying on any 3D ground-truth information. Our method achieves the state-of-the-art performance of semantic scene completion on two large-scale benchmark datasets MatterPort3D and ScanNet, surpasses baselines even with costly 3D annotations in predicting both geometry and semantics. To our knowledge, our method is also the first 2D-driven method addressing completion and semantic segmentation of real-world 3D scans simultaneously.

A System of Configurable 3D Indoor Scene Synthesis Via Semantic Relation Learning

Learning 3 D Scene Synthesis from Annotated RGB-D Images

InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior

Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions

Learning to Synthesize 3D Indoor Scenes from Monocular Images.

Language-guided Semantic Style Transfer of 3D Indoor Scenes

RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texture

Fast 3D Indoor Scene Synthesis by LearningSpatial Relation Priors of Objects

CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Embedding

SceneCraft: Layout-Guided 3D Scene Generation

SceneTeller: Language-to-3D Scene Generation

Fast 3D Indoor Scene Synthesis by Learning Spatial Relation Priors of Objects

SSR-2D: Semantic 3D Scene Reconstruction from 2D Images

3D scene generation from scene graphs and self-attention

Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars

S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field

Forest2Seq: Revitalizing Order Prior for Sequential Indoor Scene Synthesis

OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation

The Scene Language: Representing Scenes with Programs, Words, and Embeddings

SynthText3D:synthesizing Scene Text Images from 3D Virtual Worlds

SynthText3D: synthesizing scene text images from 3D virtual worlds