360-GS: Layout-guided Panoramic Gaussian Splatting For Indoor Roaming

Jiayang Bai,Letian Huang,Jie Guo,Wen Gong,Yuanqi Li,Yanwen Guo

2024-02-02

Abstract:3D Gaussian Splatting (3D-GS) has recently attracted great attention with real-time and photo-realistic renderings. This technique typically takes perspective images as input and optimizes a set of 3D elliptical Gaussians by splatting them onto the image planes, resulting in 2D Gaussians. However, applying 3D-GS to panoramic inputs presents challenges in effectively modeling the projection onto the spherical surface of ${360^\circ}$ images using 2D Gaussians. In practical applications, input panoramas are often sparse, leading to unreliable initialization of 3D Gaussians and subsequent degradation of 3D-GS quality. In addition, due to the under-constrained geometry of texture-less planes (e.g., walls and floors), 3D-GS struggles to model these flat regions with elliptical Gaussians, resulting in significant floaters in novel views. To address these issues, we propose 360-GS, a novel $360^{\circ}$ Gaussian splatting for a limited set of panoramic inputs. Instead of splatting 3D Gaussians directly onto the spherical surface, 360-GS projects them onto the tangent plane of the unit sphere and then maps them to the spherical projections. This adaptation enables the representation of the projection using Gaussians. We guide the optimization of 360-GS by exploiting layout priors within panoramas, which are simple to obtain and contain strong structural information about the indoor scene. Our experimental results demonstrate that 360-GS allows panoramic rendering and outperforms state-of-the-art methods with fewer artifacts in novel view synthesis, thus providing immersive roaming in indoor scenarios.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The paper addresses the problem of applying 3D Gaussian scattering (3D-GS) technique to panoramic images to achieve real-time high-quality panoramic rendering and new view synthesis in indoor roaming. Existing 3D-GS methods face challenges such as spatial distortion, sparse input, and difficulties in geometric modeling of planes with less texture when dealing with panoramic inputs, resulting in decreased rendering quality and artifacts such as "floating objects". To solve these problems, the paper proposes 360-GS, which is a 360-degree Gaussian scattering method that utilizes layout prior to effectively optimize 3D Gaussian distribution and reduce artifacts in new views, thus improving the panoramic rendering effect.