Abstract:This paper proposes a new method for simultaneous 3D reconstruction and semantic segmentation of indoor scenes. Unlike existing methods that require recording a video using a color camera and/or a depth camera, our method only needs a small number of (e.g., 3-5) color images from uncalibrated sparse views as input, which greatly simplifies data acquisition and extends applicable scenarios. Since different views have limited overlaps, our method allows a single image as input to discern the depth and semantic information of the scene. The key issue is how to recover relatively accurate depth from single images and reconstruct a 3D scene by fusing very few depth maps. To address this problem, we first design an iterative deep architecture, IterNet, that estimates depth and semantic segmentation alternately, so that they benefit each other. To deal with the little overlap and non-rigid transformation between views, we further propose a joint global and local registration method to reconstruct a 3D scene with semantic information from sparse views. We also make available a new indoor synthetic dataset simultaneously providing photorealistic high-resolution RGB images, accurate depth maps and pixel-level semantic labels for thousands of complex layouts, useful for training and evaluation. Experimental results on public datasets and our dataset demonstrate that our method achieves more accurate depth estimation, smaller semantic segmentation errors and better 3D reconstruction results, compared with state-of-the-art methods.

Learning to Reconstruct 3D Manhattan Wireframes from a Single Image

Learning to Parse Wireframes in Images of Man-Made Environments

Learning to Construct 3D Building Wireframes from 3D Line Clouds

Neural 3D Scene Reconstruction with the Manhattan-world Assumption

PC2WF: 3D Wireframe Reconstruction from Raw Point Clouds

NEAT: Distilling 3D Wireframes from Neural Attraction Fields

Line-based single view 3D reconstruction in Manhattan world for augmented reality

Embedding Visual Cognition in 3D Reconstruction from Multi-View Engineering Drawings

Neural 3D Scene Reconstruction with Indoor Planar Priors

Learning to Reconstruct and Understand Indoor Scenes from Sparse Views

Image-based building reconstruction with Manhattan-world assumption

HoW-3D: Holistic 3D Wireframe Perception from a Single Image

End-to-End Wireframe Parsing

City3D: Large-Scale Building Reconstruction from Airborne LiDAR Point Clouds

Generating 3D House Wireframes with Semantics

Learning to sculpt neural cityscapes

Novel Single View Constraints for Manhattan 3D Line Reconstruction

Learning Structural Graph Layouts and 3D Shapes for Long Span Bridges 3D Reconstruction

Real-time indoor scene reconstruction with Manhattan assumption

Reconstruction of 3D Information of Buildings from Single-View Images Based on Shadow Information

Data-Driven 3D Reconstruction of Dressed Humans From Sparse Views