Abstract:Benchmarking vision-based driving policies is challenging. On one hand, open-loop evaluation with real data is easy, but these results do not reflect closed-loop performance. On the other, closed-loop evaluation is possible in simulation, but is hard to scale due to its significant computational demands. Further, the simulators available today exhibit a large domain gap to real data. This has resulted in an inability to draw clear conclusions from the rapidly growing body of research on end-to-end autonomous driving. In this paper, we present NAVSIM, a middle ground between these evaluation paradigms, where we use large datasets in combination with a non-reactive simulator to enable large-scale real-world benchmarking. Specifically, we gather simulation-based metrics, such as progress and time to collision, by unrolling bird's eye view abstractions of the test scenes for a short simulation horizon. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. As we demonstrate empirically, this decoupling allows open-loop metric computation while being better aligned with closed-loop evaluations than traditional displacement errors. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights. On a large set of challenging scenarios, we observe that simple methods with moderate compute requirements such as TransFuser can match recent large-scale end-to-end driving architectures such as UniAD. Our modular framework can potentially be extended with new datasets, data curation strategies, and metrics, and will be continually maintained to host future challenges. Our code is available at <a class="link-external link-https" href="https://github.com/autonomousvision/navsim" rel="external noopener nofollow">this https URL</a>.

Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

Learning to Simulate Complex Scenes for Street Scene Segmentation

FastSim: A Modular and Plug-and-Play Simulator for Aerial Robots

HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving

GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving

GarchingSim: An Autonomous Driving Simulator with Photorealistic Scenes and Minimalist Workflow

Reconstructing Objects In-the-wild for Realistic Sensor Simulation

NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

CARLA2Real: a tool for reducing the sim2real gap in CARLA simulator

Learning Interactive Real-World Simulators

Learning autonomous driving from aerial imagery

UniSim: A Neural Closed-Loop Sensor Simulator

Realistic 3D Simulators for Automotive: A Review of Main Applications and Features

SLAV-Sim: A Framework for Self-Learning Autonomous Vehicle Simulation

A system for generating complex physically accurate sensor images for automotive applications

CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation

UnrealCV

Using simulation to quantify the performance of automotive perception systems

The Radiance of Neural Fields: Democratizing Photorealistic and Dynamic Robotic Simulation

RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator

A Gamified Simulator and Physical Platform for Self-Driving Algorithm Training and Validation