LLM4Drive: A Survey of Large Language Models for Autonomous Driving

Zhenjie Yang,Xiaosong Jia,Hongyang Li,Junchi Yan

2024-08-12

Abstract:Autonomous driving technology, a catalyst for revolutionizing transportation and urban mobility, has the tend to transition from rule-based systems to data-driven strategies. Traditional module-based systems are constrained by cumulative errors among cascaded modules and inflexible pre-set rules. In contrast, end-to-end autonomous driving systems have the potential to avoid error accumulation due to their fully data-driven training process, although they often lack transparency due to their "black box" nature, complicating the validation and traceability of decisions. Recently, large language models (LLMs) have demonstrated abilities including understanding context, logical reasoning, and generating answers. A natural thought is to utilize these abilities to empower autonomous driving. By combining LLM with foundation vision models, it could open the door to open-world understanding, reasoning, and few-shot learning, which current autonomous driving systems are lacking. In this paper, we systematically review a research line about \textit{Large Language Models for Autonomous Driving (LLM4AD)}. This study evaluates the current state of technological advancements, distinctly outlining the principal challenges and prospective directions for the field. For the convenience of researchers in academia and industry, we provide real-time updates on the latest advances in the field as well as relevant open-source resources via the designated link: <a class="link-external link-https" href="https://github.com/Thinklab-SJTU/Awesome-LLM4AD" rel="external noopener nofollow">this https URL</a>.

Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address several key issues in the field of autonomous driving:

1. **Limitations of traditional modular systems**: Traditional module-based autonomous driving systems suffer from cumulative errors and rigid preset rules between modules, leading to overall performance degradation.
2. **Transparency issues of end-to-end systems**: Although end-to-end autonomous driving systems can avoid cumulative errors between modules, their "black box" nature makes the decision-making process difficult to interpret and verify.
3. **Enhancing decision-making capabilities**: By combining large language models (LLMs) with foundational vision models, the autonomous driving system's understanding, reasoning, and few-shot learning capabilities in an open world are improved.
4. **Improving interpretability**: Utilizing the generality and strong reasoning capabilities of large language models to enhance the transparency and interpretability of autonomous driving systems.

The paper systematically reviews the current research progress on the application of large language models in autonomous driving and explores future development directions.