Abstract:This paper proposes a new method for simultaneous 3D reconstruction and semantic segmentation of indoor scenes. Unlike existing methods that require recording a video using a color camera and/or a depth camera, our method only needs a small number of (e.g., 3-5) color images from uncalibrated sparse views as input, which greatly simplifies data acquisition and extends applicable scenarios. Since different views have limited overlaps, our method allows a single image as input to discern the depth and semantic information of the scene. The key issue is how to recover relatively accurate depth from single images and reconstruct a 3D scene by fusing very few depth maps. To address this problem, we first design an iterative deep architecture, IterNet, that estimates depth and semantic segmentation alternately, so that they benefit each other. To deal with the little overlap and non-rigid transformation between views, we further propose a joint global and local registration method to reconstruct a 3D scene with semantic information from sparse views. We also make available a new indoor synthetic dataset simultaneously providing photorealistic high-resolution RGB images, accurate depth maps and pixel-level semantic labels for thousands of complex layouts, useful for training and evaluation. Experimental results on public datasets and our dataset demonstrate that our method achieves more accurate depth estimation, smaller semantic segmentation errors and better 3D reconstruction results, compared with state-of-the-art methods.

Vanishing Point Detection and Scene Reconstruction of Cluttered Room

Hybrid 3D Reconstruction of Indoor Scenes Integrating Object Recognition

3D Reconstruction of Indoor Scenes Via Image Registration

Understanding of Indoor Scenes Based on Projection of Spatial Rectangles.

The Polygonal 3D Layout Reconstruction of an Indoor Environment via Voxel-Based Room Segmentation and Space Partition

3D Residential Reconstruction from a Single Indoor Floor Plan.

Outdoor Scene 3D Reconstruction from Multiple Point Cloud

Real-time indoor scene reconstruction with Manhattan assumption

3D Indoor Scene Reconstruction and Layout Based on Virtual Reality Technology and Few-Shot Learning

Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image

Learning to Reconstruct and Understand Indoor Scenes from Sparse Views

Estimating Spatial Layout of Cluttered Rooms by Using Object Prior and Spatial Constraints

Modeling Indoor Scenes with Repetitions from 3D Raw Point Data

Online Structure Analysis for Real-Time Indoor Scene Reconstruction

Three-Dimensional Structure Measurement And Optimization Method Of Indoor Scene Based On Single Image

Three-Dimensional Reconstruction of Indoor Scenes Based on Implicit Neural Representation

3D Reconstruction of Indoor Scenes Based on 3DGS Models

Indoor Scene Reconstruction: from Panorama Images to CAD Models.

Visual Odometry Based 3D-Reconstruction

Planar Reconstruction of Indoor Scenes from Sparse Views and Relative Camera Poses

A New Era of Indoor Scene Reconstruction: A Survey