Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap

Carl Lindström,Georg Hess,Adam Lilja,Maryam Fatemi,Lars Hammarstrand,Christoffer Petersson,Lennart Svensson

2024-03-24

Abstract:Neural Radiance Fields (NeRFs) have emerged as promising tools for advancing autonomous driving (AD) research, offering scalable closed-loop simulation and data augmentation capabilities. However, to trust the results achieved in simulation, one needs to ensure that AD systems perceive real and rendered data in the same way. Although the performance of rendering methods is increasing, many scenarios will remain inherently challenging to reconstruct faithfully. To this end, we propose a novel perspective for addressing the real-to-simulated data gap. Rather than solely focusing on improving rendering fidelity, we explore simple yet effective methods to enhance perception model robustness to NeRF artifacts without compromising performance on real data. Moreover, we conduct the first large-scale investigation into the real-to-simulated data gap in an AD setting using a state-of-the-art neural rendering technique. Specifically, we evaluate object detectors and an online mapping model on real and simulated data, and study the effects of different pre-training strategies. Our results show notable improvements in model robustness to simulated data, even improving real-world performance in some cases. Last, we delve into the correlation between the real-to-simulated gap and image reconstruction metrics, identifying FID and LPIPS as strong indicators.

Robotics

What problem does this paper attempt to address?

This paper explores how to narrow the gap between the real and simulated data in the automatic driving system when processing Neural Radiance Fields (NeRF) rendering data. The study proposes that in addition to improving rendering fidelity, the perception model should also enhance its robustness to NeRF artifacts while maintaining its performance on real data. The paper conducts a large-scale investigation, evaluating the performance of object detectors and online mapping models on real and simulated data, and examines the impact of pre-training strategies. The results demonstrate that simple data augmentation techniques can significantly improve the model's robustness to simulated data and sometimes even enhance performance in the real world.

Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap

NeuRAD: Neural Rendering for Autonomous Driving

DriveEnv-NeRF: Exploration of A NeRF-Based Autonomous Driving Environment for Real-World Performance Validation

Neural Radiance Field in Autonomous Driving: A Survey

S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generation

Benchmarking Neural Radiance Fields for Autonomous Robots: An Overview

Closing the Visual Sim-to-Real Gap with Object-Composable NeRFs

Towards a Robust Framework for NeRF Evaluation

LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes

NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields

Adv3D: Generating 3D Adversarial Examples for 3D Object Detection in Driving Scenarios with NeRF

SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving

LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields

NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fields

S-NeRF: Neural Radiance Fields for Street Views

NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review

MapNeRF: Incorporating Map Priors into Neural Radiance Fields for Driving View Simulation

Transient Neural Radiance Fields for Lidar View Synthesis and 3D Reconstruction

Large-Scale Neural Scene Disentanglement Approach for Self-Driving Simulation

NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving

Enhance-NeRF: Multiple Performance Evaluation for Neural Radiance Fields