Abstract:To guarantee the safety and reliability of autonomous vehicle (AV) systems, corner cases play a crucial role in exploring the system's behavior under rare and challenging conditions within simulation environments. However, current approaches often fall short in meeting diverse testing needs and struggle to generalize to novel, high-risk scenarios that closely mirror real-world conditions. To tackle this challenge, we present AutoScenario, a multimodal Large Language Model (LLM)-based framework for realistic corner case generation. It converts safety-critical real-world data from multiple sources into textual representations, enabling the generalization of key risk factors while leveraging the extensive world knowledge and advanced reasoning capabilities of <a class="link-external link-http" href="http://LLMs.Furthermore" rel="external noopener nofollow">this http URL</a>, it integrates tools from the Simulation of Urban Mobility (SUMO) and CARLA simulators to simplify and execute the code generated by LLMs. Our experiments demonstrate that AutoScenario can generate realistic and challenging test scenarios, precisely tailored to specific testing requirements or textual descriptions. Additionally, we validated its ability to produce diverse and novel scenarios derived from multimodal real-world data involving risky situations, harnessing the powerful generalization capabilities of LLMs to effectively simulate a wide range of corner cases.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of how to generate realistic and diverse corner cases when the autonomous vehicle (AV) system explores its behavior in the simulation environment. Specifically, the paper focuses on the following points: 1. **Limitations of existing methods**: - Current methods are difficult to meet diverse testing requirements and perform poorly in generalizing to novel, high - risk scenarios that require more real - world - like conditions. - Existing data - driven and rule - based methods usually rely on pre - collected data or initial conditions, resulting in the generated scenarios lacking diversity and being less realistic. 2. **Importance of safety - critical scenarios**: - Discovering and testing corner cases is crucial for ensuring the safety of autonomous vehicles and accelerating the development cycle. As autonomous driving technology progresses, corner cases become increasingly rare and diverse, so defining and identifying the most relevant corner cases becomes more critical. 3. **Inadequacy of control mechanisms**: - Currently, there is a lack of effective control mechanisms to flexibly generate scenarios according to abstract requirements. Developers usually conceive scenarios in a broad conceptual way, while simulation requires detailed configurations such as road geometry and precise vehicle positions. To solve these problems, the paper proposes **AutoScenario**, a framework based on the multimodal large - language model (LLM) for automatically generating realistic corner cases. It generalizes key risk factors by converting safety - critical real - world data from multiple sources into text representations and leveraging the powerful reasoning ability and extensive world knowledge of the LLM. In addition, it integrates SUMO and CARLA simulator tools to simplify and execute the code generated by the LLM, in order to improve the authenticity and stability of the generated scenarios. ### Summary The main objective of the paper is to automate the generation of realistic and diverse safety - critical scenarios by combining multimodal inputs and the capabilities of large - language models to ensure the safety and reliability of autonomous vehicle systems. This not only addresses the lack of diversity and realism in existing methods but also provides an efficient and controllable solution to cope with the complex and changing traffic environment.

Realistic Corner Case Generation for Autonomous Vehicles with Multimodal Large Language Model

Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles

Corner Case Generation and Analysis for Safety Assessment of Autonomous Vehicles

Generating Out-Of-Distribution Scenarios Using Language Models

LeGEND: A Top-Down Approach to Scenario Generation of Autonomous Driving Systems Assisted by Large Language Models

ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles

Generating Edge Cases for Testing Autonomous Vehicles Using Real-World Data

Heterogeneous Driver Modeling and Corner Scenarios Sampling for Automated Vehicles Testing

Scenario Generation for Autonomous Vehicles with Deep-Learning-Based Heterogeneous Driver Models: Implementation and Verification

Empowering Corner Case Detection in Autonomous Vehicles with Multimodal Large Language Models

Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation

Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priors

Empowering Autonomous Driving with Large Language Models: A Safety Perspective

Chat2Scenario: Scenario Extraction From Dataset Through Utilization of Large Language Model

Generating and Explaining Corner Cases Using Learnt Probabilistic Lane Graphs

SurrealDriver: Designing Generative Driver Agent Simulation Framework in Urban Contexts based on Large Language Model

Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles

Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models

CC-SGG: Corner Case Scenario Generation using Learned Scene Graphs

Adversarial Generation of Safety-Critical Lane-Change Scenarios for Autonomous Vehicles

Generating Critical Scenarios for Testing Automated Driving Systems