RaNeuS: Ray-adaptive Neural Surface Reconstruction

Yida Wang,David Joseph Tan,Nassir Navab,Federico Tombari

2024-06-14

Abstract:Our objective is to leverage a differentiable radiance field \eg NeRF to reconstruct detailed 3D surfaces in addition to producing the standard novel view renderings. There have been related methods that perform such tasks, usually by utilizing a signed distance field (SDF). However, the state-of-the-art approaches still fail to correctly reconstruct the small-scale details, such as the leaves, ropes, and textile surfaces. Considering that different methods formulate and optimize the projection from SDF to radiance field with a globally constant Eikonal regularization, we improve with a ray-wise weighting factor to prioritize the rendering and zero-crossing surface fitting on top of establishing a perfect SDF. We propose to adaptively adjust the regularization on the signed distance field so that unsatisfying rendering rays won't enforce strong Eikonal regularization which is ineffective, and allow the gradients from regions with well-learned radiance to effectively back-propagated to the SDF. Consequently, balancing the two objectives in order to generate accurate and detailed surfaces. Additionally, concerning whether there is a geometric bias between the zero-crossing surface in SDF and rendering points in the radiance field, the projection becomes adjustable as well depending on different 3D locations during optimization. Our proposed \textit{RaNeuS} are extensively evaluated on both synthetic and real datasets, achieving state-of-the-art results on both novel view synthesis and geometric reconstruction.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main goal of this paper is to improve existing neural surface reconstruction methods to enhance the accuracy of recovering fine 3D surfaces of static scenes or objects from multi-view images. Specifically, the researchers aim to address the following issues:

1. **Detail Reconstruction Issue**: Existing techniques perform poorly in reconstructing small-scale details (such as leaves, ropes, and textile surfaces). The paper proposes a new method to improve the reconstruction of these details.

2. **Balancing Rendering and Geometric Reconstruction**: The study finds that during the optimization process, the optimization speed of the surface distance field (SDF) may be faster than that of the radiance field, limiting the impact of the radiance field on the reconstruction. Therefore, the paper focuses on finding a method to balance the optimization between appearance (i.e., rendering quality) and geometry to achieve high-quality surface reconstruction.

3. **Adaptive Regularization**: The paper proposes two adaptive weight factors (\(\lambda_r\) and \(\lambda_g\)) to adjust the Eikonal regularization of the SDF. These factors can dynamically adjust based on rendering quality and geometric deviation, thereby improving the reconstruction of thin structures and reducing the geometric deviation between the SDF surface and the radiance field.

With the above improvements, the method proposed in the paper (RaNeuS) demonstrates superior performance over existing methods across multiple datasets, particularly in recovering fine details. Experimental results show that RaNeuS not only improves rendering quality (evaluated by PSNR and SSIM metrics) but also significantly enhances the accuracy of 3D geometric reconstruction (evaluated by Chamfer distance).