Abstract:Recent studies have highlighted the promising application of NeRF in autonomous driving contexts. However, the complexity of outdoor environments, combined with the restricted viewpoints in driving scenarios, complicates the task of precisely reconstructing scene geometry. Such challenges often lead to diminished quality in reconstructions and extended durations for both training and rendering. To tackle these challenges, we present Lightning NeRF. It uses an efficient hybrid scene representation that effectively utilizes the geometry prior from LiDAR in autonomous driving scenarios. Lightning NeRF significantly improves the novel view synthesis performance of NeRF and reduces computational overheads. Through evaluations on real-world datasets, such as KITTI-360, Argoverse2, and our private dataset, we demonstrate that our approach not only exceeds the current state-of-the-art in novel view synthesis quality but also achieves a five-fold increase in training speed and a ten-fold improvement in rendering speed. Codes are available at https://github.com/VISION-SJTU/Lightning-NeRF .

What problem does this paper attempt to address?

This paper attempts to solve several key problems encountered when applying NeRF (Neural Radiance Fields) for novel view synthesis in autonomous driving scenarios: 1. **Accurate Reconstruction in Complex Environments**: The complexity of outdoor environments, combined with the limited viewing angles in driving scenarios, makes it difficult to accurately reconstruct the scene geometry. These problems usually lead to a decline in reconstruction quality and an increase in training and rendering time. 2. **Computational Efficiency**: Traditional NeRF methods are slow in training and rendering and have a huge computational cost when dealing with large - scale outdoor scenes because they need to query the deep MLP (Multi - Layer Perceptron) millions of times. 3. **Background Modeling**: When dealing with unbounded outdoor scenes, how to effectively model the background part to avoid burdening the representation of the foreground part is an important challenge. To solve these problems, the paper proposes **Lightning NeRF**, an efficient hybrid scene representation method. Specifically, Lightning NeRF solves the above problems in the following ways: - **Hybrid Scene Representation**: Model density and color separately, using explicit and implicit methods. The density part is explicitly modeled using a voxel grid with a limited resolution, eliminating the need for MLP; the color part retains the implicitly modeled MLP to adapt to the highly variable real world. - **LiDAR Initialization**: Use LiDAR point cloud data to initialize the scene geometry, which significantly reduces the representational challenges and improves the convergence speed and rendering efficiency of the model. - **Color Decomposition**: Decompose the color into two parts, view - dependent and view - independent, and model them using lightweight MLPs respectively. This not only improves the performance of the model on unseen views but also enhances the extrapolation ability of the model. Through these improvements, Lightning NeRF has been evaluated on multiple real - world autonomous driving datasets, and the results show that it exceeds the current state - of - the - art methods in the quality of novel view synthesis, and improves the training speed and rendering speed by 5 times and 10 times respectively.

Lightning NeRF: Efficient Hybrid Scene Representation for Autonomous Driving

HarmonicNeRF: Geometry-Informed Synthetic View Augmentation for 3D Scene Reconstruction in Driving Scenarios

S-NeRF: Neural Radiance Fields for Street Views

Large-Scale Neural Scene Disentanglement Approach for Self-Driving Simulation

PC-NeRF: Parent-Child Neural Radiance Fields Using Sparse LiDAR Frames in Autonomous Driving Environments

LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields

PC-NeRF: Parent-Child Neural Radiance Fields under Partial Sensor Data Loss in Autonomous Driving Environments

LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes

DriveEnv-NeRF: Exploration of A NeRF-Based Autonomous Driving Environment for Real-World Performance Validation

Camera and LiDAR Fusion for Urban Scene Reconstruction and Novel View Synthesis via Voxel-Based Neural Radiance Fields

LarsNeRF: Fast and Effective View Synthesis from Low-Altitude Remote Sensing Scenarios

S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generation

Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations

Drone-NeRF: Efficient NeRF based 3D scene reconstruction for large-scale drone survey

Collaborative neural radiance fields for novel view synthesis

LiDeNeRF: Neural radiance field reconstruction with depth prior provided by LiDAR point cloud

NeuRAD: Neural Rendering for Autonomous Driving

Multi-Level Neural Scene Graphs for Dynamic Urban Environments

Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes

Neural Rendering based Urban Scene Reconstruction for Autonomous Driving

NeRFVS: Neural Radiance Fields for Free View Synthesis Via Geometry Scaffolds