Abstract:We present ChatScene, a Large Language Model (LLM)-based agent that leverages the capabilities of LLMs to generate safety-critical scenarios for autonomous vehicles. Given unstructured language instructions, the agent first generates textually described traffic scenarios using LLMs. These scenario descriptions are subsequently broken down into several sub-descriptions for specified details such as behaviors and locations of vehicles. The agent then distinctively transforms the textually described sub-scenarios into domain-specific languages, which then generate actual code for prediction and control in simulators, facilitating the creation of diverse and complex scenarios within the CARLA simulation environment. A key part of our agent is a comprehensive knowledge retrieval component, which efficiently translates specific textual descriptions into corresponding domain-specific code snippets by training a knowledge database containing the scenario description and code pairs. Extensive experimental results underscore the efficacy of ChatScene in improving the safety of autonomous vehicles. For instance, the scenarios generated by ChatScene show a 15% increase in collision rates compared to state-of-the-art baselines when tested against different reinforcement learning-based ego vehicles. Furthermore, we show that by using our generated safety-critical scenarios to fine-tune different RL-based autonomous driving models, they can achieve a 9% reduction in collision rates, surpassing current SOTA methods. ChatScene effectively bridges the gap between textual descriptions of traffic scenarios and practical CARLA simulations, providing a unified way to conveniently generate safety-critical scenarios for safety testing and improvement for AVs.

LASER: Script Execution by Autonomous Agents for On-demand Traffic Simulation

LASER: Script Execution by Autonomous Agents for On-demand Traffic Simulation

SurrealDriver: Designing Generative Driver Agent Simulation Framework in Urban Contexts based on Large Language Model

Scalable Traffic Simulation for Autonomous Driving Via Multi-Agent Goal Assignment and Autoregressive Goal-Directed Planning

A Language Agent for Autonomous Driving

Deep Reinforcement Learning for Adverse Garage Scenario Generation

AADS: Augmented autonomous driving simulation using data-driven algorithms

SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Data

A Novel Traffic Simulation Framework for Testing Autonomous Vehicles Using SUMO and CARLA

ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles

LASIL: Learner-Aware Supervised Imitation Learning For Long-term Microscopic Traffic Simulation

Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model

LASER: LLM Agent with State-Space Exploration for Web Navigation

LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs

Driving Everywhere with Large Language Model Policy Adaptation

LCSim: A Large-Scale Controllable Traffic Simulator

LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving

Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models

CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving

DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving