Abstract:We, as human beings, can understand and picture a familiar scene from arbitrary viewpoints given a single image, whereas this is still a grand challenge for computers. We hereby present a novel solution to mimic such human perception capability based on a new paradigm of a modal 3D scene understanding with neural rendering for a closed scene. Specifically, we first learn the prior knowledge of the objects in a closed scene via an offline stage, which facilitates an online stage to understand the room with unseen furniture arrangement. During the online stage, given a panoramic image of the scene in different layouts, we utilize a holistic neural-rendering-based optimization framework to efficiently estimate the correct 3D scene layout and deliver realistic free-viewpoint rendering. In order to handle the domain gap between the offline and online stage, our method exploits compositional neural rendering techniques for data augmentation in the offline training. The experiments on both synthetic and real datasets demonstrate that our two-stage design achieves robust 3D scene understanding and outperforms competing methods by a large margin, and we also show that our realistic free-viewpoint rendering enables various applications, including scene touring and editing. Code and data are available on the project webpage: https://zju3dv.github.io/nr_in_a_room/.

3D Room Layout Estimation Beyond the Manhattan World Assumption

3D Room Layout Estimation from a Single RGB Image.

3D Room Layout Estimation from a Cubemap of Panorama Image Via Deep Manhattan Hough Transform

LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image

Self-supervised 360$^{\circ}$ Room Layout Estimation

General 3D Room Layout from a Single View by Render-and-Compare

Manhattan Room Layout Reconstruction from a Single $360^{\circ }$ Image: A Comparative Study of State-of-the-Art Methods.

LED2-Net: Monocular 360 Layout Estimation via Differentiable Depth Rendering

Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects

Transferable End-to-end Room Layout Estimation via Implicit Encoding

Estimating Generic 3D Room Structures from 2D Annotations

Spatial Geometric Reasoning for Room Layout Estimation via Deep Reinforcement Learning

DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama

Layouts from Panoramic Images with Geometry and Deep Learning

3D Indoor Scene Reconstruction and Layout Based on Virtual Reality Technology and Few-Shot Learning

Self-training Room Layout Estimation via Geometry-aware Ray-casting

No More Ambiguity in 360° Room Layout via Bi-Layout Estimation

GeoLayout: Geometry Driven Room Layout Estimation Based on Depth Maps of Planes

Estimating spatial layout of rooms from RGB-D videos.

From Semi-supervised to Omni-supervised Room Layout Estimation Using Point Clouds.

The Polygonal 3D Layout Reconstruction of an Indoor Environment via Voxel-Based Room Segmentation and Space Partition