Abstract:Reconstructing 3D layouts from multiple $360^{\circ}$ panoramas has received increasing attention recently as estimating a complete layout of a large-scale and complex room from a single panorama is very difficult. The state-of-the-art method, called PSMNet, introduces the first learning-based framework that jointly estimates the room layout and registration given a pair of panoramas. However, PSMNet relies on an approximate (i.e., "noisy") registration as input. Obtaining this input requires a solution for wide baseline registration which is a challenging problem. In this work, we present a complete multi-view panoramic layout estimation framework that jointly learns panorama registration and layout estimation given a pair of panoramas without relying on a pose prior. The major improvement over PSMNet comes from a novel Geometry-aware Panorama Registration Network or GPR-Net that effectively tackles the wide baseline registration problem by exploiting the layout geometry and computing fine-grained correspondences on the layout boundaries, instead of the global pixel-space. Our architecture consists of two parts. First, given two panoramas, we adopt a vision transformer to learn a set of 1D horizon features sampled on the panorama. These 1D horizon features encode the depths of individual layout boundary samples and the correspondence and covisibility maps between layout boundaries. We then exploit a non-linear registration module to convert these 1D horizon features into a set of corresponding 2D boundary points on the layout. Finally, we estimate the final relative camera pose via RANSAC and obtain the complete layout simply by taking the union of registered layouts. Experimental results indicate that our method achieves state-of-the-art performance in both panorama registration and layout estimation on a large-scale indoor panorama dataset ZInD.

PanoAnnotator

DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization

Automatic 3D Indoor Scene Modeling from Single Panorama

Pano2Room: Novel View Synthesis from a Single Indoor Panorama

Manhattan Room Layout Reconstruction from a Single $360^{\circ }$ Image: A Comparative Study of State-of-the-Art Methods.

Modeling room structure from indoor panorama.

LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image

Estimating Generic 3D Room Structures from 2D Annotations

Smartannotator an Interactive Tool for Annotating Indoor Rgbd Images

Indoor Scene Reconstruction: from Panorama Images to CAD Models.

PanoViT: Vision Transformer for Room Layout Estimation from a Single Panoramic Image

Layout-Guided Indoor Panorama Inpainting with Plane-Aware Normalization

Indoor Structure Understanding from Single 360 Cylindrical Panoramic Image

360 Layout Estimation via Orthogonal Planes Disentanglement and Multi-view Geometric Consistency Perception

GPR-Net: Multi-view Layout Estimation via a Geometry-aware Panorama Registration Network

Efficient 3D Room Shape Recovery from a Single Panorama.

Layouts from Panoramic Images with Geometry and Deep Learning

PanGEA: The Panoramic Graph Environment Annotation Toolkit

3D Room Layout Estimation from a Cubemap of Panorama Image Via Deep Manhattan Hough Transform

Disentangling Orthogonal Planes for Indoor Panoramic Room Layout Estimation with Cross-Scale Distortion Awareness