HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving

Hongyu Zhou,Longzhong Lin,Jiabao Wang,Yichong Lu,Dongfeng Bai,Bingbing Liu,Yue Wang,Andreas Geiger,Yiyi Liao

2024-12-03

Abstract:In the past few decades, autonomous driving algorithms have made significant progress in perception, planning, and control. However, evaluating individual components does not fully reflect the performance of entire systems, highlighting the need for more holistic assessment methods. This motivates the development of HUGSIM, a closed-loop, photo-realistic, and real-time simulator for evaluating autonomous driving algorithms. We achieve this by lifting captured 2D RGB images into the 3D space via 3D Gaussian Splatting, improving the rendering quality for closed-loop scenarios, and building the closed-loop environment. In terms of rendering, We tackle challenges of novel view synthesis in closed-loop scenarios, including viewpoint extrapolation and 360-degree vehicle rendering. Beyond novel view synthesis, HUGSIM further enables the full closed simulation loop, dynamically updating the ego and actor states and observations based on control commands. Moreover, HUGSIM offers a comprehensive benchmark across more than 70 sequences from KITTI-360, Waymo, nuScenes, and PandaSet, along with over 400 varying scenarios, providing a fair and realistic evaluation platform for existing autonomous driving algorithms. HUGSIM not only serves as an intuitive evaluation benchmark but also unlocks the potential for fine-tuning autonomous driving algorithms in a photorealistic closed-loop setting.

Computer Vision and Pattern Recognition,Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the limitations of existing autonomous driving algorithm evaluation methods, especially that the open - loop evaluation method cannot comprehensively reflect the performance of the entire system. Specifically:

1. **Limitations of Open - Loop Evaluation**: Existing evaluation methods are mainly based on collected data sets to conduct open - loop evaluations of specific components (such as perception and planning) in autonomous driving systems. However, this method cannot capture the performance in closed - loop scenarios. In particular, evaluations conducted in pre - collected safe and normal driving environments cannot fully reflect the reliability and effectiveness of autonomous driving algorithms in complex and critical scenarios.

2. **Requirement for Closed - Loop Simulators**: To overcome the above limitations, it is necessary to develop advanced closed - loop simulators that can safely explore beyond the normal range and critical safety scenarios in virtual environments, thereby comprehensively evaluating the performance of autonomous driving systems.

For this purpose, the paper proposes HUGSIM, a real - time, highly realistic closed - loop simulator for evaluating autonomous driving algorithms. The main features of HUGSIM include:

- **High - Fidelity Rendering**: Through 3D Gaussian Splatting technology, the captured 2D RGB images are lifted into 3D space, improving the rendering quality in closed - loop scenarios.
- **Viewpoint Extrapolation and 360 - Degree Vehicle Rendering**: It addresses the challenges in new - view synthesis, including viewpoint extrapolation and 360 - degree vehicle rendering.
- **Dynamic Update**: It realizes a full - closed - loop simulation environment that can dynamically update the states and observations of the ego - vehicle and actors according to control commands.
- **Comprehensive Benchmarking**: It provides comprehensive benchmarking of more than 70 sequences and 400 different scenarios from multiple data sets, providing a fair and realistic evaluation platform for existing autonomous driving algorithms.

Through these features, HUGSIM not only serves as an intuitive evaluation benchmark but also unlocks the potential for fine - tuning autonomous driving algorithms in a highly realistic closed - loop setting.