Abstract:Autonomous Driving (AD) encounters significant safety hurdles in long-tail unforeseen driving scenarios, largely stemming from the non-interpretability and poor generalization of the deep neural networks within the AD system, particularly in out-of-distribution and uncertain data. To this end, this paper explores the integration of Large Language Models (LLMs) into AD systems, leveraging their robust common-sense knowledge and reasoning abilities. The proposed methodologies employ LLMs as intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning, for enhancing driving performance and safety. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine. Demonstrating superior performance and safety metrics compared to state-of-the-art approaches, our approach shows the promising potential for using LLMs for autonomous vehicles.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the significant safety obstacles of autonomous driving (AD) systems in long - tailed unforeseen driving scenarios. These problems mainly stem from the inexplicability and poor generalization ability of deep neural networks (DNNs), especially when dealing with out - of - distribution data and uncertain data. To solve these problems, the paper explores integrating large language models (LLMs) into AD systems, making use of their strong common - sense knowledge and reasoning abilities. Specifically, the paper proposes the following points: 1. **Intelligent decision - making in behavior planning**: By using LLMs as intelligent decision - makers in behavior planning, enhance driving performance and safety. 2. **Safety verifier for situational safety learning**: Introduce a safety verifier to ensure that the decisions of LLMs meet safety requirements and conduct real - time learning and adjustment. The paper presents two key research cases: - **Model predictive control (MPC) under adaptive LLM conditions**: Combine the decisions of LLMs with low - level MPC to optimize trajectory planning and ensure safety. - **Interactive behavior planning scheme**: Through the state machine framework, realize the interactive multi - step decision - making between LLMs and surrounding vehicles, and improve the performance and safety of driving tasks. These methods aim to improve the safety and performance of autonomous driving systems, especially in complex and unforeseen driving environments. ### Formula Examples Some of the formulas involved in the paper include: - Vehicle dynamics model: \[ s_{t + 1}=f(s_t, u_t) \] where \( s=(x, y, v_x, v_y)\in S\subset\mathbb{R}^4 \), representing longitudinal position, lateral position, longitudinal velocity and lateral velocity respectively; \( u_t\in U\subset\mathbb{R}^2 \) represents continuous control inputs (such as acceleration and steering signals). - Safety constraint conditions: \[ |x_i-\hat{x}_j^i|-L\geq0,\quad|x_i - \bar{\hat{x}}_j^i|-L\geq0,\quad\text{where Lane}([y_j^i,\bar{y}_j^i]) == \text{Lane}(y_i) \] These formulas are used to describe the state update and safety constraint conditions of vehicles, ensuring that the autonomous driving system can make safe and reliable decisions in complex traffic environments.

Empowering Autonomous Driving with Large Language Models: A Safety Perspective

A Survey on Large Language Model-empowered Autonomous Driving

LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

Large Language Models for Autonomous Driving (LLM4AD): Concept, Benchmark, Simulation, and Real-Vehicle Experiment

Drive Like a Human: Rethinking Autonomous Driving with Large Language Models

Large Language Models for Human-like Autonomous Driving: A Survey

LLM4Drive: A Survey of Large Language Models for Autonomous Driving

Facilitating Autonomous Driving Tasks with Large Language Models

DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving

DriveLLM: Charting the Path Toward Full Autonomous Driving with Large Language Models

Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles

A Survey on Multimodal Large Language Models for Autonomous Driving

Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles

VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision

Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles

SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Models

Evaluation of Large Language Models for Decision Making in Autonomous Driving

Driving Everywhere with Large Language Model Policy Adaptation

A Superalignment Framework in Autonomous Driving with Large Language Models

Large Language Model Safety: A Holistic Survey

XLM for Autonomous Driving Systems: A Comprehensive Review