Abstract:This paper introduces CRITICAL, a novel closed-loop framework for autonomous vehicle (AV) training and testing. CRITICAL stands out for its ability to generate diverse scenarios, focusing on critical driving situations that target specific learning and performance gaps identified in the Reinforcement Learning (RL) agent. The framework achieves this by integrating real-world traffic dynamics, driving behavior analysis, surrogate safety measures, and an optional Large Language Model (LLM) component. It is proven that the establishment of a closed feedback loop between the data generation pipeline and the training process can enhance the learning rate during training, elevate overall system performance, and augment safety resilience. Our evaluations, conducted using the Proximal Policy Optimization (PPO) and the HighwayEnv simulation environment, demonstrate noticeable performance improvements with the integration of critical case generation and LLM analysis, indicating CRITICAL's potential to improve the robustness of AV systems and streamline the generation of critical scenarios. This ultimately serves to hasten the development of AV agents, expand the general scope of RL training, and ameliorate validation efforts for AV safety.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **Enhance the ability of autonomous vehicles (AVs) to handle critical driving scenarios during training and testing, in order to improve their performance, safety, and adaptability**. Specifically, traditional autonomous vehicle training methods and environments often involve controllable and repeatable scenarios. Although these scenarios are helpful for initial learning, they may not fully cover the complexity of the real world. Therefore, after being trained in such an environment, a reinforcement - learning (RL) - based model may not be able to cope well with the unpredictable dynamics in road traffic. In addition, accurately replicating the diverse conditions in real - world driving and addressing the specific failure modes that autonomous vehicles may encounter are also significant challenges. To solve these problems, the paper proposes a new framework named **CRITICAL**, which aims to enhance conventional RL training by automatically generating diverse critical driving scenarios. This framework combines real - world traffic dynamics, driving behavior analysis, alternative safety measures, and large - language - model (LLM) components to generate more challenging and diverse training scenarios. This not only improves training efficiency but also enhances the overall performance and safety of the autonomous driving system. ### Main contributions: 1. **Introduction of the CRITICAL framework**: This framework increases the exposure of RL agents to various scenarios, especially critical driving scenarios, thereby enhancing the performance and resilience of autonomous vehicles. 2. **Utilization of the highD dataset**: Analyze and replicate diverse driving behaviors through clustering techniques, and use risk measures to create high - fidelity critical scenarios, enriching the simulation environment. 3. **Empirical research verification**: Research shows that the closed - loop feedback mechanism of CRITICAL significantly improves the learning rate, overall AV performance, and the adaptability to safety - critical situations. Through these improvements, the CRITICAL framework can accelerate the development of autonomous driving agents, expand the overall scope of RL training, and improve the verification work of AV safety.

Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation

Adaptive Deep Reinforcement Learning for Critical Boundary Scenario Generation

Continual Driving Policy Optimization with Closed-Loop Individualized Curricula

Dynamically Expanding Capacity of Autonomous Driving with Near-Miss Focused Training Framework

CRASH: Challenging Reinforcement-Learning Based Adversarial Scenarios For Safety Hardening

ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles

Adversarial Generation of Safety-Critical Lane-Change Scenarios for Autonomous Vehicles

Stackelberg Driver Model for Continual Policy Improvement in Scenario-Based Closed-Loop Autonomous Driving

Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles

Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles

Game-Theoretic Adversarial Interaction-Based Critical Scenario Generation for Autonomous Vehicles

Empowering Autonomous Driving with Large Language Models: A Safety Perspective

Realistic Corner Case Generation for Autonomous Vehicles with Multimodal Large Language Model

Generating Critical Scenarios for Testing Automated Driving Systems

Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles

Autonomous Highway Driving using Deep Reinforcement Learning

Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)

Think2Drive: Efficient Reinforcement Learning by Thinking with Latent World Model for Autonomous Driving (in CARLA-v2)

Evolving testing scenario generation and intelligence evaluation for automated vehicles

Generating Out-Of-Distribution Scenarios Using Language Models

Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priors