Abstract:Rigorously testing autonomy systems is essential for making safe self-driving vehicles (SDV) a reality. It requires one to generate safety critical scenarios beyond what can be collected safely in the world, as many scenarios happen rarely on public roads. To accurately evaluate performance, we need to test the SDV on these scenarios in closed-loop, where the SDV and other actors interact with each other at each timestep. Previously recorded driving logs provide a rich resource to build these new scenarios from, but for closed loop evaluation, we need to modify the sensor data based on the new scene configuration and the SDV's decisions, as actors might be added or removed and the trajectories of existing actors and the SDV will differ from the original log. In this paper, we present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle and converts it into a realistic closed-loop multi-sensor simulation. UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene, and composites them together to simulate LiDAR and camera data at new viewpoints, with actors added or removed and at new placements. To better handle extrapolated views, we incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions. Our experiments show UniSim can simulate realistic sensor data with small domain gap on downstream tasks. With UniSim, we demonstrate closed-loop evaluation of an autonomy system on safety-critical scenarios as if it were in the real world.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to generate realistic, multi - sensor, closed - loop simulation data in the development of self - driving vehicles (SDV) to achieve safe and efficient testing of self - driving systems. Specifically, the paper points out the following deficiencies in current self - driving system testing methods: 1. **Log Replay**: This method does not allow the self - driving system to perform actions and observe their effects because the new sensor data is different from the original record. 2. **Driving more miles in the real world**: This method is neither safe nor scalable and sustainable. To solve these problems, the paper proposes a neural sensor simulator named UniSim. The main goal of UniSim is to generate realistic, multi - modal sensor simulation data from a single recorded log and support the modification of scene configurations (such as adding or removing dynamic objects, changing the ego - vehicle trajectory, etc.), thereby achieving closed - loop self - driving system evaluation. ### Main Contributions 1. **High - Fidelity Multi - Sensor Simulation**: UniSim can generate realistic camera and LiDAR observation data from a single recorded log, supporting new perspectives and scene configurations. 2. **Scene Editing Capability**: UniSim allows independent modeling and manipulation of static backgrounds and dynamic objects, thereby generating new scene configurations. 3. **Closed - Loop Simulation Evaluation**: UniSim can evaluate the performance of self - driving systems in a closed - loop environment, simulating safety - critical scenarios in the real world without actual driving. ### Technical Details - **Neural Feature Fields**: Used to represent the geometric structures and appearance features of static backgrounds and dynamic objects. - **Multi - resolution Feature Grids**: Improve feature representation ability and inference speed. - **Convolutional Network**: Used to process feature maps in new perspectives and generate the final image. - **Dynamic Object Priors**: Render unseen areas by learning shape priors. ### Experimental Results - **Image Simulation**: UniSim performs well in both interpolation and lane - offset tests, and especially in the extrapolation setting, its performance is significantly better than other methods. - **LiDAR Simulation**: The LiDAR point clouds generated by UniSim have higher fidelity, less noise, and better continuity. - **Detection Domain Gap**: The simulation data generated by UniSim shows a smaller domain gap in downstream tasks (such as object detection, motion prediction, and planning). In conclusion, UniSim provides a powerful tool that can generate realistic multi - sensor simulation data, support the closed - loop evaluation of self - driving systems, and thus promote the safe and efficient development of self - driving technology.

UniSim: A Neural Closed-Loop Sensor Simulator

NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving

AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles

HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving

SLAV-Sim: A Framework for Self-Learning Autonomous Vehicle Simulation

Sim-on-Wheels: Physical World in the Loop Simulation for Self-Driving

Diagnosing and Predicting Autonomous Vehicle Operational Safety Using Multiple Simulation Modalities and a Virtual Environment

UniSim: An Autonomous Multi-Agent Simulation Method with Intelligent Perception

A Pioneering Scalable Self-driving Car Simulation Platform

Closed-Loop Simulation Test Method for Autonomous Vehicles Based on Adversarial Scenarios

3D Environmental Perception Modeling in the Simulated Autonomous-Driving Systems

OASim: an Open and Adaptive Simulator based on Neural Rendering for Autonomous Driving

Learning Naturalistic Driving Environment with Statistical Realism

Simulating Dynamic Driving Behavior in Simulation Test for Unmanned Vehicles via Multi-Sensor Data.

LimSim: A Long-term Interactive Multi-scenario Traffic Simulator

CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation

Learning Interactive Real-World Simulators

Customized Co-Simulation Environment for Autonomous Driving Algorithm Development and Evaluation

A Novel Traffic Simulation Framework for Testing Autonomous Vehicles Using SUMO and CARLA

Autonomous Driving Simulation for Unmanned Vehicles.