ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles

Jiawei Zhang,Chejian Xu,Bo Li

2024-05-23

Abstract:We present ChatScene, a Large Language Model (LLM)-based agent that leverages the capabilities of LLMs to generate safety-critical scenarios for autonomous vehicles. Given unstructured language instructions, the agent first generates textually described traffic scenarios using LLMs. These scenario descriptions are subsequently broken down into several sub-descriptions for specified details such as behaviors and locations of vehicles. The agent then distinctively transforms the textually described sub-scenarios into domain-specific languages, which then generate actual code for prediction and control in simulators, facilitating the creation of diverse and complex scenarios within the CARLA simulation environment. A key part of our agent is a comprehensive knowledge retrieval component, which efficiently translates specific textual descriptions into corresponding domain-specific code snippets by training a knowledge database containing the scenario description and code pairs. Extensive experimental results underscore the efficacy of ChatScene in improving the safety of autonomous vehicles. For instance, the scenarios generated by ChatScene show a 15% increase in collision rates compared to state-of-the-art baselines when tested against different reinforcement learning-based ego vehicles. Furthermore, we show that by using our generated safety-critical scenarios to fine-tune different RL-based autonomous driving models, they can achieve a 9% reduction in collision rates, surpassing current SOTA methods. ChatScene effectively bridges the gap between textual descriptions of traffic scenarios and practical CARLA simulations, providing a unified way to conveniently generate safety-critical scenarios for safety testing and improvement for AVs.

Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The paper aims to address the issue of safety testing for autonomous vehicles (AVs) when faced with various complex and hazardous driving scenarios. Specifically:

- **Generating Safety-Critical Scenarios**: By leveraging the capabilities of large language models (LLMs), the paper proposes a new method called ChatScene for generating safety-critical scenarios for autonomous vehicles. This method can generate detailed traffic scene descriptions based on natural language instructions and convert them into actual code for simulation in the CARLA environment.
- **Improving Testing Efficiency and Diversity**: Traditional field testing is not only costly but also requires the collection of a large amount of data to cover all possible safety-critical scenarios. Therefore, the paper emphasizes the importance of generating test scenarios using simulation environments and demonstrates the advantages of ChatScene in generating diverse and complex scenarios.
- **Enhancing Autonomous Driving Safety**: Experimental results show that scenarios generated by ChatScene can significantly increase the collision rate (by 15% compared to existing technologies), but by fine-tuning the autonomous driving model using these scenarios, the collision rate can be effectively reduced (by 9%). This demonstrates the potential of ChatScene in enhancing the robustness of autonomous driving systems.

In summary, this research aims to develop an efficient and flexible method to generate safety-critical scenarios, thereby improving the safety and reliability testing process of autonomous vehicles.