Neuralangelo: High-Fidelity Neural Surface Reconstruction

Zhaoshuo Li,Thomas Müller,Alex Evans,Russell H. Taylor,Mathias Unberath,Ming-Yu Liu,Chen-Hsuan Lin

2023-06-13

Abstract:Neural surface reconstruction has been shown to be powerful for recovering dense 3D surfaces via image-based neural rendering. However, current methods struggle to recover detailed structures of real-world scenes. To address the issue, we present Neuralangelo, which combines the representation power of multi-resolution 3D hash grids with neural surface rendering. Two key ingredients enable our approach: (1) numerical gradients for computing higher-order derivatives as a smoothing operation and (2) coarse-to-fine optimization on the hash grids controlling different levels of details. Even without auxiliary inputs such as depth, Neuralangelo can effectively recover dense 3D surface structures from multi-view images with fidelity significantly surpassing previous methods, enabling detailed large-scale scene reconstruction from RGB video captures.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the accuracy and detail level of 3D surface reconstruction from multi - view images. Specifically, current methods have difficulties in restoring the detailed structures of real - world scenes. To address this challenge, the paper proposes the Neuralangelo framework, which combines the representational power of multi - resolution 3D hash grids and neural surface rendering techniques to achieve high - fidelity 3D surface reconstruction and can work effectively even without auxiliary data such as depth or segmentation.

### Main Contributions
1. **Framework Innovation**: The Neuralangelo framework naturally incorporates the representational power of multi - resolution hash encoding into the neural SDF (Signed Distance Function) representation.
2. **Technical Improvements**: Two simple but effective techniques are proposed to improve the quality of hash - encoding - based surface reconstruction:
- **Higher - Order Derivatives and Numerical Gradients**: Numerical gradients are used to calculate higher - order derivatives (such as surface normals for eikonal regularization), which are crucial for stabilizing the optimization process.
- **Coarse - to - Fine Optimization Strategy**: Different levels of detail are gradually optimized to effectively restore different levels of structures.
3. **Experimental Verification**: Through extensive experiments on standard benchmarks and real - world scenes, significant improvements in reconstruction accuracy and view synthesis quality of Neuralangelo are demonstrated.

### Technical Details
- **Neural Volume Rendering**: NeRF (Neural Radiance Field) maps 3D spatial positions to colors and volume densities through an MLP and uses neural volume rendering to combine these predicted values into pixel colors.
- **SDF Representation**: SDF implicitly represents a surface \( S=\{x\in\mathbb{R}^3\mid f(x) = 0\} \) through its zero - level set, where \( f(x) \) is the SDF value.
- **Multi - Resolution Hash Encoding**: Using a multi - resolution grid, the corner points of each grid cell are mapped to a hash entry to store the encoded features. The input position \( x_i \) is mapped to a position \( x_{i,l}=x_i\cdot V_l \) on each grid resolution \( V_l \) and the feature vector \( \gamma_l(x_{i,l}) \) is obtained through trilinear interpolation.

### Key Technologies
- **Numerical Gradient Calculation**: Analytic gradients are discontinuous in position, causing optimization updates to only propagate to the local hash grid and lack non - local smoothness. By using numerical gradients, it is possible to go beyond local hash grid cells and achieve smoother optimization.
- **Gradual Optimization**: By gradually activating hash grids of different resolutions, the problem of "re - learning" of fine - grained grids during the coarse optimization process is avoided, thus better capturing details.

### Experimental Results
- **DTU Benchmark**: Neuralangelo achieves the best performance in both Chamfer distance and PSNR metrics, even without using auxiliary inputs.
- **Tanks and Temples Dataset**: In large - scale indoor/outdoor scenes, Neuralangelo performs best in PSNR and F1 - score and can restore high - fidelity surfaces with complex details.

### Conclusion
Neuralangelo significantly improves the quality and detail level of 3D surface reconstruction from multi - view images through simple technological innovations without relying on auxiliary data such as depth or segmentation. This makes Neuralangelo widely applicable and advantageous in practical applications.