Large Language Models for Human-like Autonomous Driving: A Survey

Yun Li,Kai Katsumata,Ehsan Javanmardi,Manabu Tsukada

2024-07-27

Abstract:Large Language Models (LLMs), AI models trained on massive text corpora with remarkable language understanding and generation capabilities, are transforming the field of Autonomous Driving (AD). As AD systems evolve from rule-based and optimization-based methods to learning-based techniques like deep reinforcement learning, they are now poised to embrace a third and more advanced category: knowledge-based AD empowered by LLMs. This shift promises to bring AD closer to human-like AD. However, integrating LLMs into AD systems poses challenges in real-time inference, safety assurance, and deployment costs. This survey provides a comprehensive and critical review of recent progress in leveraging LLMs for AD, focusing on their applications in modular AD pipelines and end-to-end AD systems. We highlight key advancements, identify pressing challenges, and propose promising research directions to bridge the gap between LLMs and AD, thereby facilitating the development of more human-like AD systems. The survey first introduces LLMs' key features and common training schemes, then delves into their applications in modular AD pipelines and end-to-end AD, respectively, followed by discussions on open challenges and future directions. Through this in-depth analysis, we aim to provide insights and inspiration for researchers and practitioners working at the intersection of AI and autonomous vehicles, ultimately contributing to safer, smarter, and more human-centric AD technologies.

Artificial Intelligence,Robotics

What problem does this paper attempt to address?

The paper aims to explore how large language models (LLMs) can be applied to autonomous driving (AD) systems to achieve decision-making and control that more closely resemble human driving behavior. Specifically, the paper addresses the following major issues:

1. **Technological Evolution**: It reviews the evolution of autonomous driving systems, from rule-based methods and optimization methods to learning-based technologies, and proposes a new direction of enhancing autonomous driving by using LLMs as knowledge bases and reasoning engines.

2. **Modular Decision-Making and End-to-End Systems**: It provides a detailed analysis of the application of LLMs in modular decision-making processes, such as improving decision quality by integrating multimodal inputs like visual data and sensor information. It also discusses how end-to-end autonomous driving systems can utilize LLMs for holistic perception, planning, and control.

3. **Challenges and Future Directions**: It identifies the main challenges currently faced in applying LLMs to autonomous driving, including real-time inference speed, safety validation, interpretability, and potential social biases. The paper also proposes future research directions aimed at further optimizing algorithm performance, enhancing model safety and robustness, and ultimately achieving more intelligent, safe, and human-like autonomous driving technology.

In summary, this paper aims to provide researchers and practitioners with a comprehensive understanding framework by reviewing recent research achievements of LLMs in the field of autonomous driving, thereby promoting the development of this interdisciplinary field.