Abstract:In the realm of autonomous driving, achieving precise 3D reconstruction of the driving environment is critical for ensuring safety and effective navigation. Neural Radiance Fields (NeRF) have shown promise in creating highly detailed and accurate models of complex environments. However, the application of NeRF in autonomous driving scenarios encounters several challenges, primarily due to the sparsity of viewpoints inherent in camera trajectories and the constraints on data collection in unbounded outdoor scenes, which typically occur along predetermined paths. This limitation not only reduces the available scene information but also poses significant challenges for NeRF training, as the sparse and path-distributed observational data leads to under-representation of the scene's geometry. In this paper, we introduce HarmonicNeRF, a novel approach for outdoor self-supervised monocular scene reconstruction. HarmonicNeRF capitalizes on the strengths of NeRF and enhances surface reconstruction accuracy by augmenting the input space with geometry-informed synthetic views. This is achieved through the application of spherical harmonics to generate novel radiance values, taking into careful consideration the color observations from the limited available real-world views. Additionally, our method incorporates proxy geometry to effectively manage occlusion, generating radiance pseudo-labels that circumvent the limitations of traditional image-warping techniques, which often fail in sparse data conditions typical of autonomous driving environments. Extensive experiments conducted on the KITTI, Argoverse, and NuScenes datasets demonstrate our approach establishes new benchmarks in synthesizing novel depth views and reconstructing scenes, significantly outperforming existing methods. Project page: <a class="link-external link-https" href="https://github.com/Jiawei-Yao0812/HarmonicNeRF" rel="external noopener nofollow">this https URL</a>

A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction

AG-NeRF: Attention-guided Neural Radiance Fields for Multi-height Large-scale Outdoor Scene Rendering

Crowd-Sourced NeRF: Collecting Data from Production Vehicles for 3D Street View Reconstruction

ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objects

Large-Scale Indoor Visual-Geometric Multimodal Dataset and Benchmark for Novel View Synthesis

Aerial-NeRF: Adaptive Spatial Partitioning and Sampling for Large-Scale Aerial Rendering

NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes

TextNeRF: A Novel Scene-Text Image Synthesis Method Based on Neural Radiance Fields

DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision

Collaborative neural radiance fields for novel view synthesis

SC-NeRF: Self-Correcting Neural Radiance Field with Sparse Views

Enhance-NeRF: Multiple Performance Evaluation for Neural Radiance Fields

StructNeRF: Neural Radiance Fields for Indoor Scenes With Structural Hints

Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields

NeRF synthesis with shading guidance

SG-NeRF: Neural Surface Reconstruction with Scene Graph Optimization

Benchmarking Neural Radiance Fields for Autonomous Robots: An Overview

Improving Neural Radiance Fields with Depth-aware Optimization for Novel View Synthesis

HarmonicNeRF: Geometry-Informed Synthetic View Augmentation for 3D Scene Reconstruction in Driving Scenarios

Block-NeRF: Scalable Large Scene Neural View Synthesis

MD-NeRF: Enhancing Large-Scale Scene Rendering and Synthesis With Hybrid Point Sampling and Adaptive Scene Decomposition