NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance

Hanlin Chen,Chen Li,Gim Hee Lee

2023-12-01

Abstract:Existing neural implicit surface reconstruction methods have achieved impressive performance in multi-view 3D reconstruction by leveraging explicit geometry priors such as depth maps or point clouds as regularization. However, the reconstruction results still lack fine details because of the over-smoothed depth map or sparse point cloud. In this work, we propose a neural implicit surface reconstruction pipeline with guidance from 3D Gaussian Splatting to recover highly detailed surfaces. The advantage of 3D Gaussian Splatting is that it can generate dense point clouds with detailed structure. Nonetheless, a naive adoption of 3D Gaussian Splatting can fail since the generated points are the centers of 3D Gaussians that do not necessarily lie on the surface. We thus introduce a scale regularizer to pull the centers close to the surface by enforcing the 3D Gaussians to be extremely thin. Moreover, we propose to refine the point cloud from 3D Gaussians Splatting with the normal priors from the surface predicted by neural implicit models instead of using a fixed set of points as guidance. Consequently, the quality of surface reconstruction improves from the guidance of the more accurate 3D Gaussian splatting. By jointly optimizing the 3D Gaussian Splatting and the neural implicit model, our approach benefits from both representations and generates complete surfaces with intricate details. Experiments on Tanks and Temples verify the effectiveness of our proposed method.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper aims to address the issue of insufficient detail in multi-view 3D surface reconstruction. Specifically, existing neural implicit surface reconstruction methods achieve good multi-view 3D reconstruction results by leveraging explicit geometric priors such as depth maps or point clouds, but the reconstructed results still lack fine details, mainly due to overly smooth depth maps or sparse point clouds. To solve this problem, the paper proposes a new framework called NeuSG, which combines 3D Gaussian Splatting technology.

The main contributions of NeuSG include:
1. **Joint Optimization Framework**: By jointly optimizing the neural implicit model NeuS with 3D Gaussian Splatting, using the dense point clouds generated by 3D Gaussian Splatting to guide the training of NeuS, and further optimizing the results of 3D Gaussian Splatting with the normals predicted by NeuS.
2. **Scale Regularization**: Introducing a scale regularization term that makes the 3D Gaussian distribution extremely flat, thereby pulling its center points closer to the surface.
3. **Normal Regularization**: Further refining the positions of the point clouds by aligning the smallest scale direction of the 3D Gaussian distribution to the surface normals, to achieve more accurate surface reconstruction.

Experimental results show that NeuSG achieves higher surface reconstruction quality than existing methods on multiple benchmark datasets (such as Tanks and Temples), particularly excelling in detail preservation. Additionally, NeuSG also has advantages in computational efficiency, being able to complete high-quality reconstruction tasks in a shorter amount of time.