Phys3DGS: Physically-based 3D Gaussian Splatting for Inverse Rendering

Euntae Choi,Sungjoo Yoo

2024-09-16

Abstract:We propose two novel ideas (adoption of deferred rendering and mesh-based representation) to improve the quality of 3D Gaussian splatting (3DGS) based inverse rendering. We first report a problem incurred by hidden Gaussians, where Gaussians beneath the surface adversely affect the pixel color in the volume rendering adopted by the existing methods. In order to resolve the problem, we propose applying deferred rendering and report new problems incurred in a naive application of deferred rendering to the existing 3DGS-based inverse rendering. In an effort to improve the quality of 3DGS-based inverse rendering under deferred rendering, we propose a novel two-step training approach which (1) exploits mesh extraction and utilizes a hybrid mesh-3DGS representation and (2) applies novel regularization methods to better exploit the mesh. Our experiments show that, under relighting, the proposed method offers significantly better rendering quality than the existing 3DGS-based inverse rendering methods. Compared with the SOTA voxel grid-based inverse rendering method, it gives better rendering quality while offering real-time rendering.

Graphics,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issues encountered in existing relightable 3D Gaussian Splatting (3DGS) methods during inverse rendering, particularly the decline in rendering quality under new lighting conditions. Specifically:

1. **Hidden Gaussian Points Issue**: The paper points out that in existing relightable 3DGS methods, Gaussian points located below the surface (i.e., hidden Gaussian points) affect pixel colors, leading to a decline in rendering quality. This issue is especially pronounced in the early stages of training when opacity has not been fully learned.

2. **Proposed Method**: To overcome this issue, the authors propose a physically-based 3D Gaussian Splatting method (Phys3DGS) based on deferred rendering. By adopting deferred rendering techniques, the paper addresses the problem caused by hidden Gaussian points and introduces a new two-step training method that leverages a hybrid grid-3DGS representation to improve shape learning. Additionally, some novel regularization methods are proposed to enhance the quality of geometric structure learning.

3. **Experimental Results**: Experiments show that compared to existing relightable 3DGS methods, this method significantly improves rendering quality under new lighting conditions and outperforms traditional voxel grid inverse rendering methods in real-time rendering.

In summary, the paper focuses on improving the performance of existing 3DGS methods under complex lighting conditions and achieves higher rendering quality and faster rendering speed through a series of technical innovations.