GS-Octree: Octree-based 3D Gaussian Splatting for Robust Object-level 3D Reconstruction Under Strong Lighting

Jiaze Li,Zhengyu Wen,Luo Zhang,Jiangbei Hu,Fei Hou,Zhebin Zhang,Ying He

2024-06-26

Abstract:The 3D Gaussian Splatting technique has significantly advanced the construction of radiance fields from multi-view images, enabling real-time rendering. While point-based rasterization effectively reduces computational demands for rendering, it often struggles to accurately reconstruct the geometry of the target object, especially under strong lighting. To address this challenge, we introduce a novel approach that combines octree-based implicit surface representations with Gaussian splatting. Our method consists of four stages. Initially, it reconstructs a signed distance field (SDF) and a radiance field through volume rendering, encoding them in a low-resolution octree. The initial SDF represents the coarse geometry of the target object. Subsequently, it introduces 3D Gaussians as additional degrees of freedom, which are guided by the SDF. In the third stage, the optimized Gaussians further improve the accuracy of the SDF, allowing it to recover finer geometric details compared to the initial SDF obtained in the first stage. Finally, it adopts the refined SDF to further optimize the 3D Gaussians via splatting, eliminating those that contribute little to visual appearance. Experimental results show that our method, which leverages the distribution of 3D Gaussians with SDFs, reconstructs more accurate geometry, particularly in images with specular highlights caused by strong lighting.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the challenges encountered in object-level 3D reconstruction based on multi-view images under strong lighting conditions. Specifically, existing methods struggle to accurately reconstruct the geometric structure of the target object when dealing with specular highlights caused by strong lighting. Although point-based methods can reduce computational demands during the rendering process, they perform poorly in reconstructing the geometric details of the target object.

To tackle this challenge, the authors propose a novel approach that combines Octree representation with Gaussian splatting. This method is implemented through the following four stages:

1. **Initial Reconstruction**: First, a Signed Distance Field (SDF) and a radiance field are reconstructed through Volume Rendering and encoded in a low-resolution Octree, thereby obtaining a rough geometric shape of the target object.
2. **Introduction of Gaussian Points**: Next, 3D Gaussian points are introduced as additional degrees of freedom, which are optimized guided by the SDF.
3. **Refinement of SDF**: The optimized Gaussian points further improve the accuracy of the SDF, allowing the SDF to recover more details than the initial SDF, thus overcoming the impact of specular highlights.
4. **Optimization of Gaussian Points**: Finally, the refined SDF is used to further optimize the 3D Gaussian points, eliminating Gaussian points that contribute little to the visual appearance.

Experimental results show that this method can more accurately reconstruct the geometric structure of the target object under strong lighting conditions, particularly excelling in images with a large number of specular highlights. Additionally, this method can significantly reduce the number of required Gaussian points while maintaining high-quality rendering effects.