Abstract:We, as human beings, can understand and picture a familiar scene from arbitrary viewpoints given a single image, whereas this is still a grand challenge for computers. We hereby present a novel solution to mimic such human perception capability based on a new paradigm of a modal 3D scene understanding with neural rendering for a closed scene. Specifically, we first learn the prior knowledge of the objects in a closed scene via an offline stage, which facilitates an online stage to understand the room with unseen furniture arrangement. During the online stage, given a panoramic image of the scene in different layouts, we utilize a holistic neural-rendering-based optimization framework to efficiently estimate the correct 3D scene layout and deliver realistic free-viewpoint rendering. In order to handle the domain gap between the offline and online stage, our method exploits compositional neural rendering techniques for data augmentation in the offline training. The experiments on both synthetic and real datasets demonstrate that our two-stage design achieves robust 3D scene understanding and outperforms competing methods by a large margin, and we also show that our realistic free-viewpoint rendering enables various applications, including scene touring and editing. Code and data are available on the project webpage: https://zju3dv.github.io/nr_in_a_room/.

3D Room Layout Estimation from a Single RGB Image.

Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image

LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image

WallNet: Reconstructing General Room Layouts from RGB Images

3D Room Layout Estimation Beyond the Manhattan World Assumption

Bottom-up Estimation of Geometric Layout for Indoor Images

Estimating spatial layout of rooms from RGB-D videos.

Edge-Semantic Learning Strategy for Layout Estimation in Indoor Environment

General 3D Room Layout from a Single View by Render-and-Compare

Self-supervised 360$^{\circ}$ Room Layout Estimation

Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects

Layouts from Panoramic Images with Geometry and Deep Learning

Estimating Generic 3D Room Structures from 2D Annotations

3D Residential Reconstruction from a Single Indoor Floor Plan.

RoomStructNet: Learning to Rank Non-Cuboidal Room Layouts From Single View

From Edge to Keypoint: An End-to-End Framework For Indoor Layout Estimation

DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama

Indoor Layout Estimation by 2D LiDAR and Camera Fusion

Low-Drift RGB-D SLAM with Room Reconstruction Using Scene Understanding

Transferable End-to-end Room Layout Estimation via Implicit Encoding

Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image