Abstract:The deployment of autonomous vehicles controlled by machine learning techniques requires extensive testing in diverse real-world environments, robust handling of edge cases and out-of-distribution scenarios, and comprehensive safety validation to ensure that these systems can navigate safely and effectively under unpredictable conditions. Addressing Out-Of-Distribution (OOD) driving scenarios is essential for enhancing safety, as OOD scenarios help validate the reliability of the models within the vehicle's autonomy stack. However, generating OOD scenarios is challenging due to their long-tailed distribution and rarity in urban driving dataset. Recently, Large Language Models (LLMs) have shown promise in autonomous driving, particularly for their zero-shot generalization and common-sense reasoning capabilities. In this paper, we leverage these LLM strengths to introduce a framework for generating diverse OOD driving scenarios. Our approach uses LLMs to construct a branching tree, where each branch represents a unique OOD scenario. These scenarios are then simulated in the CARLA simulator using an automated framework that aligns scene augmentation with the corresponding textual descriptions. We evaluate our framework through extensive simulations, and assess its performance via a diversity metric that measures the richness of the scenarios. Additionally, we introduce a new "OOD-ness" metric, which quantifies how much the generated scenarios deviate from typical urban driving conditions. Furthermore, we explore the capacity of modern Vision-Language Models (VLMs) to interpret and safely navigate through the simulated OOD scenarios. Our findings offer valuable insights into the reliability of language models in addressing OOD scenarios within the context of urban driving.

Towards Scenario Retrieval of Real Driving Data with Large Vision-Language Models

Chat2Scenario: Scenario Extraction From Dataset Through Utilization of Large Language Model

Embodied Understanding of Driving Scenarios

Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models

VLP: Vision Language Planning for Autonomous Driving

DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

DriveLM: Driving with Graph Visual Question Answering

Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model

Generating Out-Of-Distribution Scenarios Using Language Models

Evaluation of Large Language Models for Decision Making in Autonomous Driving

Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Driving

Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles

Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving

Personalized Autonomous Driving with Large Language Models: Field Experiments

Vision Language Models in Autonomous Driving: A Survey and Outlook

Probing Multimodal LLMs as World Models for Driving

UnstrPrompt: Large Language Model Prompt for Driving in Unstructured Scenarios

VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenes

Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles

BEV-TSR: Text-Scene Retrieval in BEV Space for Autonomous Driving

V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models