LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

Hao Sha,Yao Mu,Yuxuan Jiang,Li Chen,Chenfeng Xu,Ping Luo,Shengbo Eben Li,Masayoshi Tomizuka,Wei Zhan,Mingyu Ding

2023-10-14

Abstract:Existing learning-based autonomous driving (AD) systems face challenges in comprehending high-level information, generalizing to rare events, and providing interpretability. To address these problems, this work employs Large Language Models (LLMs) as a decision-making component for complex AD scenarios that require human commonsense understanding. We devise cognitive pathways to enable comprehensive reasoning with LLMs, and develop algorithms for translating LLM decisions into actionable driving commands. Through this approach, LLM decisions are seamlessly integrated with low-level controllers by guided parameter matrix adaptation. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination, thanks to the commonsense reasoning capabilities of LLMs. This paper presents an initial step toward leveraging LLMs as effective decision-makers for intricate AD scenarios in terms of safety, efficiency, generalizability, and interoperability. We aspire for it to serve as inspiration for future research in this field. Project page: <a class="link-external link-https" href="https://sites.google.com/view/llm-mpc" rel="external noopener nofollow">this https URL</a>

Robotics,Artificial Intelligence,Computation and Language,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

### The Problems This Paper Attempts to Solve

This paper primarily aims to address several core issues faced by existing learning-based Autonomous Driving (AD) systems:

1. **Ability to Understand High-Level Information**:
- Current autonomous driving systems lack the ability to understand complex traffic rules and common sense, especially in scenarios such as un-signalized intersections.

2. **Generalization to Rare Events**:
- Present learning-based autonomous driving systems perform poorly in handling rare or long-tail scenarios, mainly due to limited datasets and low sampling efficiency.

3. **Explainability**:
- Existing learning-based autonomous driving systems lack transparency and cannot clearly demonstrate their decision-making processes, making them difficult to be accepted by society and regulatory bodies.

To address these issues, this paper proposes a method that utilizes Large Language Models (LLMs) as the decision-making component of autonomous driving systems. Through this approach, LLMs can perform complex reasoning and translate decisions into specific driving instructions. Experiments show that this method not only outperforms baseline methods in single-vehicle tasks but also excels in complex scenarios such as multi-vehicle coordination. This research aims to demonstrate the potential of LLMs as efficient decision-makers in autonomous driving and provide a reference for future studies.