How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence

Xuemin Hu,Shen Li,Tingyu Huang,Bo Tang,Rouxing Huai,Long Chen

2023-06-30

Abstract:Safety and cost are two important concerns for the development of autonomous driving technologies. From the academic research to commercial applications of autonomous driving vehicles, sufficient simulation and real world testing are required. In general, a large scale of testing in simulation environment is conducted and then the learned driving knowledge is transferred to the real world, so how to adapt driving knowledge learned in simulation to reality becomes a critical issue. However, the virtual simulation world differs from the real world in many aspects such as lighting, textures, vehicle dynamics, and agents' behaviors, etc., which makes it difficult to bridge the gap between the virtual and real worlds. This gap is commonly referred to as the reality gap (RG). In recent years, researchers have explored various approaches to address the reality gap issue, which can be broadly classified into three categories: transferring knowledge from simulation to reality (sim2real), learning in digital twins (DTs), and learning by parallel intelligence (PI) technologies. In this paper, we consider the solutions through the sim2real, DTs, and PI technologies, and review important applications and innovations in the field of autonomous driving. Meanwhile, we show the state-of-the-arts from the views of algorithms, models, and simulators, and elaborate the development process from sim2real to DTs and PI. The presentation also illustrates the far-reaching effects and challenges in the development of sim2real, DTs, and PI in autonomous driving.

Robotics,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in the development of autonomous driving technology, how to effectively transfer the learned knowledge to the real world through a large number of tests and training in the simulation environment, so as to improve the performance of autonomous vehicles in the real environment. Specifically, the paper focuses on how to overcome the differences between virtual simulation and the real world, namely the so - called "Reality Gap (RG)". These differences include differences in lighting, texture, vehicle dynamics, agent behavior, etc., which make it difficult to directly transfer from the simulation environment to the real world. To meet this challenge, the paper explores three main methods:

1. **Transfer from Simulation to Reality (Sim2Real)**: This method focuses on improving the algorithm level. By using techniques such as Curriculum Learning, Meta - Learning, Knowledge Distillation, Robust Reinforcement Learning, Domain Randomization and Transfer Learning, the gap between simulation and reality is reduced.

2. **Digital Twins (DTs)**: The digital twin method aims to construct a mapping of real - world physical entities in the simulation environment through sensor data and physical models, thereby reflecting the entire life - cycle process of physical entities. In autonomous driving, digital twins are usually used for multi - scale modeling of the environment and vehicles. Through real - time data updates and interactions between the virtual and the real, the model is continuously optimized.

3. **Parallel Intelligence (PI)**: Parallel intelligence is a more advanced technology that combines the advantages of Sim2Real and DT methods and can better manage and control complex systems. PI has three functions: descriptive intelligence, predictive intelligence and prescriptive intelligence. By constructing an artificial system corresponding to the physical system, knowledge learning and feedback are realized.

Through a review of these methods, the paper provides a comprehensive understanding of the current research progress and discusses future research directions and challenges.