Abstract:This paper proposes a new method for simultaneous 3D reconstruction and semantic segmentation of indoor scenes. Unlike existing methods that require recording a video using a color camera and/or a depth camera, our method only needs a small number of (e.g., 3-5) color images from uncalibrated sparse views as input, which greatly simplifies data acquisition and extends applicable scenarios. Since different views have limited overlaps, our method allows a single image as input to discern the depth and semantic information of the scene. The key issue is how to recover relatively accurate depth from single images and reconstruct a 3D scene by fusing very few depth maps. To address this problem, we first design an iterative deep architecture, IterNet, that estimates depth and semantic segmentation alternately, so that they benefit each other. To deal with the little overlap and non-rigid transformation between views, we further propose a joint global and local registration method to reconstruct a 3D scene with semantic information from sparse views. We also make available a new indoor synthetic dataset simultaneously providing photorealistic high-resolution RGB images, accurate depth maps and pixel-level semantic labels for thousands of complex layouts, useful for training and evaluation. Experimental results on public datasets and our dataset demonstrate that our method achieves more accurate depth estimation, smaller semantic segmentation errors and better 3D reconstruction results, compared with state-of-the-art methods.

WallNet: Reconstructing General Room Layouts from RGB Images

Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image

LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image

3D Room Layout Estimation from a Single RGB Image.

FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans

RoomStructNet: Learning to Rank Non-Cuboidal Room Layouts From Single View

General 3D Room Layout from a Single View by Render-and-Compare

Estimating spatial layout of rooms from RGB-D videos.

3D Room Layout Estimation from a Cubemap of Panorama Image Via Deep Manhattan Hough Transform

FRI-Net: Floorplan Reconstruction via Room-wise Implicit Representation

DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama

Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects

Learning Indoor Layouts from Simple Point-Clouds

The Polygonal 3D Layout Reconstruction of an Indoor Environment via Voxel-Based Room Segmentation and Space Partition

Manhattan Room Layout Reconstruction from a Single $360^{\circ }$ Image: A Comparative Study of State-of-the-Art Methods.

Edge-Semantic Learning Strategy for Layout Estimation in Indoor Environment

Self-supervised 360$^{\circ}$ Room Layout Estimation

From Edge to Keypoint: An End-to-End Framework For Indoor Layout Estimation

Learning to Reconstruct and Understand Indoor Scenes from Sparse Views

DeepPerimeter: Indoor Boundary Estimation from Posed Monocular Sequences

Layouts from Panoramic Images with Geometry and Deep Learning