Abstract:We consider the problem of physically-based inverse rendering using 3D Gaussian Splatting (3DGS) representations. While recent 3DGS methods have achieved remarkable results in novel view synthesis (NVS), accurately capturing high-fidelity geometry, physically interpretable materials and lighting remains challenging, as it requires precise geometry modeling to provide accurate surface normals, along with physically-based rendering (PBR) techniques to ensure correct material and lighting disentanglement. Previous 3DGS methods resort to approximating surface normals, but often struggle with noisy local geometry, leading to inaccurate normal estimation and suboptimal material-lighting decomposition. In this paper, we introduce GeoSplatting, a novel hybrid representation that augments 3DGS with explicit geometric guidance and differentiable PBR equations. Specifically, we bridge isosurface and 3DGS together, where we first extract isosurface mesh from a scalar field, then convert it into 3DGS points and formulate PBR equations for them in a fully differentiable manner. In GeoSplatting, 3DGS is grounded on the mesh geometry, enabling precise surface normal modeling, which facilitates the use of PBR frameworks for material decomposition. This approach further maintains the efficiency and quality of NVS from 3DGS while ensuring accurate geometry from the isosurface. Comprehensive evaluations across diverse datasets demonstrate the superiority of GeoSplatting, consistently outperforming existing methods both quantitatively and qualitatively.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the geometric modeling and the decoupling of materials and lighting in **Physically - Based Inverse Rendering**. Specifically, the paper focuses on how to use the 3D Gaussian Splatting (3DGS) representation to achieve high - quality Novel View Synthesis (NVS), while maintaining accurate geometric shapes, physically interpretable materials and lighting. Although existing 3DGS methods have achieved remarkable results in Novel View Synthesis, they still face challenges in accurately capturing high - fidelity geometric shapes, physically interpretable materials and lighting. The main reason is that accurate geometric modeling is required to provide accurate surface normals and combine with Physically - Based Rendering (PBR) techniques to ensure the correct decoupling of materials and lighting. However, existing 3DGS methods usually deal with these problems by approximating surface normals, which leads to noise in complex local geometric structures, thus affecting the accuracy of normal estimation and the effect of material - lighting decomposition.
To overcome these challenges, the paper proposes **GeoSplatting**, a new hybrid representation method that enhances 3DGS through explicit geometric guidance and differentiable PBR equations. Specifically, GeoSplatting combines isosurfaces and 3DGS. First, it extracts an isosurface mesh from a scalar field, then converts it into 3DGS points, and formulates PBR equations for them in a fully differentiable manner. In this way, 3DGS is based on mesh geometry and can achieve accurate surface normal modeling, thereby facilitating material decomposition in the PBR framework. This method not only maintains the efficiency and quality of 3DGS in Novel View Synthesis but also ensures the accurate geometric shapes provided by isosurfaces. Comprehensive evaluations show that GeoSplatting outperforms existing methods on multiple datasets and performs excellently in both quantitative and qualitative indicators.
### Key Contributions
1. **Accurate Geometric Modeling**: By combining 3DGS with isosurface techniques, GeoSplatting can provide more accurate surface normal modeling, which is crucial for inverse rendering tasks.
2. **Efficient PBR Framework**: An efficient and differentiable PBR framework is designed, which uses the split - sum model to render 3DGS points, ensuring physically accurate material and lighting decomposition.
3. **End - to - End Optimization**: The entire process is fully differentiable and can be trained end - to - end, reducing training time and improving the quality of inverse rendering.
### Experimental Results
The paper verifies the effectiveness of GeoSplatting through extensive experiments, including Novel View Synthesis, material decomposition and relighting tasks. The experimental results show that GeoSplatting performs excellently on multiple datasets, especially in Novel View Synthesis and relighting, significantly outperforming existing methods. For example, in the Novel View Synthesis task on the NeRF dataset, GeoSplatting achieves the best performance in metrics such as PSNR, SSIM and LPIPS, and also performs well in training efficiency. In addition, GeoSplatting also shows superior performance in material decomposition and relighting tasks, especially in dealing with complex non - Lambertian materials and indirect lighting effects.