HarmonicNeRF: Geometry-Informed Synthetic View Augmentation for 3D Scene Reconstruction in Driving Scenarios

Xiaochao Pan,Jiawei Yao,Hongrui Kou,Tong Wu,Canran Xiao

2024-07-25

Abstract:In the realm of autonomous driving, achieving precise 3D reconstruction of the driving environment is critical for ensuring safety and effective navigation. Neural Radiance Fields (NeRF) have shown promise in creating highly detailed and accurate models of complex environments. However, the application of NeRF in autonomous driving scenarios encounters several challenges, primarily due to the sparsity of viewpoints inherent in camera trajectories and the constraints on data collection in unbounded outdoor scenes, which typically occur along predetermined paths. This limitation not only reduces the available scene information but also poses significant challenges for NeRF training, as the sparse and path-distributed observational data leads to under-representation of the scene's geometry. In this paper, we introduce HarmonicNeRF, a novel approach for outdoor self-supervised monocular scene reconstruction. HarmonicNeRF capitalizes on the strengths of NeRF and enhances surface reconstruction accuracy by augmenting the input space with geometry-informed synthetic views. This is achieved through the application of spherical harmonics to generate novel radiance values, taking into careful consideration the color observations from the limited available real-world views. Additionally, our method incorporates proxy geometry to effectively manage occlusion, generating radiance pseudo-labels that circumvent the limitations of traditional image-warping techniques, which often fail in sparse data conditions typical of autonomous driving environments. Extensive experiments conducted on the KITTI, Argoverse, and NuScenes datasets demonstrate our approach establishes new benchmarks in synthesizing novel depth views and reconstructing scenes, significantly outperforming existing methods. Project page: <a class="link-external link-https" href="https://github.com/Jiawei-Yao0812/HarmonicNeRF" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to address the challenges faced by 3D environment reconstruction in the autonomous driving scenario. Specifically, NeRF (Neural Radiance Fields) encounters the following problems when dealing with autonomous driving data:

1. **Sparse viewpoints**: The camera trajectories in the autonomous driving scenario are usually linear and path - restricted, resulting in very limited viewpoints and insufficient viewpoint information for high - quality synthesis.
2. **Dynamic objects and illumination changes**: There are moving objects in the autonomous driving scenario, which will cause the illumination and geometric structure to change over time, thus generating artifacts in the synthesized image and reducing the reconstruction effect.
3. **Limited data collection**: Since the autonomous driving environment is usually an unbounded outdoor scene, data collection along a predetermined path often leads to sparse data, which poses a great challenge to NeRF training.

To solve these problems, the paper introduces HarmonicNeRF, a novel self - supervised monocular scene reconstruction method. HarmonicNeRF enhances the surface reconstruction accuracy in the following ways:

- **Geometry - guided view enhancement**: Use spherical harmonic functions to generate new radiance values and combine color observations from limited real views to generate synthetic views.
- **Proxy geometry to manage occlusion**: By introducing proxy geometry to effectively handle the occlusion problem and generate radiance pseudo - labels, thus avoiding the failure of traditional image warping techniques under sparse data conditions.

Through these improvements, HarmonicNeRF can significantly improve the synthesis of new depth views and scene reconstruction performance on autonomous driving datasets such as KITTI, Argoverse, and NuScenes, surpassing existing methods.

### Main contributions of the paper

1. **Application of spherical harmonic functions**: Comprehensively integrate all color observations through spherical harmonic functions to generate pseudo - labels consistent with the natural radiance distribution, enhancing the consistency and accuracy of scene illumination and texture.
2. **Proxy geometry to handle occlusion**: Introduce proxy geometry to deal with the occlusion challenges in the ray - enhancement process, ensuring that the point radiance is not disturbed and reconstructing a more accurate surface from sparse viewpoints.
3. **Plug - in solution**: Design a general plug - in solution that is compatible with existing sparse implicit neural surface reconstruction techniques, achieving excellent sparse - view reconstruction effects in driving scenarios without the need for additional data or extensive pre - training.

### Experimental verification

The experimental results show that HarmonicNeRF has significant advantages in dealing with sparse viewpoints, especially in predicting the radiance values of new views. By comparing with the traditional linear interpolation method, HarmonicNeRF obtains a lower MSE (mean - squared error) in the quantitative evaluation on the KITTI dataset and shows better smoothness and information content in the visualization results.

In summary, HarmonicNeRF successfully solves the key challenges of 3D reconstruction in the autonomous driving scenario through innovative geometry - guided ray - enhancement techniques and spherical harmonic function fitting.