Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles

Can Cui,Yunsheng Ma,Xu Cao,Wenqian Ye,Ziran Wang

DOI: https://doi.org/10.48550/arXiv.2310.08034

2023-10-12

Abstract:The fusion of human-centric design and artificial intelligence (AI) capabilities has opened up new possibilities for next-generation autonomous vehicles that go beyond transportation. These vehicles can dynamically interact with passengers and adapt to their preferences. This paper proposes a novel framework that leverages Large Language Models (LLMs) to enhance the decision-making process in autonomous vehicles. By utilizing LLMs' linguistic and contextual understanding abilities with specialized tools, we aim to integrate the language and reasoning capabilities of LLMs into autonomous vehicles. Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks, to explore LLMs' interpretation, interaction, and reasoning in various scenarios. We also examine real-time personalization, demonstrating how LLMs can influence driving behaviors based on verbal commands. Our empirical results highlight the substantial advantages of utilizing chain-of-thought prompting, leading to improved driving decisions, and showing the potential for LLMs to enhance personalized driving experiences through ongoing verbal feedback. The proposed framework aims to transform autonomous vehicle operations, offering personalized support, transparent decision-making, and continuous learning to enhance safety and effectiveness. We achieve user-centric, transparent, and adaptive autonomous driving ecosystems supported by the integration of LLMs into autonomous vehicles.

Human-Computer Interaction,Artificial Intelligence,Robotics

What problem does this paper attempt to address?

The paper attempts to address the issue of exploring how to integrate large language models (LLMs) into autonomous vehicles to enhance the vehicle's decision-making capabilities, interactivity, and personalization levels. Specifically, the research aims to:

1. **Enhance Decision-Making Capabilities**: Improve the decision-making process of autonomous vehicles in various scenarios by leveraging the language understanding and reasoning abilities of LLMs.
2. **Achieve Natural Language Interaction**: Enable autonomous vehicles to interact with drivers through natural language, facilitating more intuitive and human-like communication.
3. **Context Understanding and Reasoning**: Enhance the vehicle's understanding of information such as traffic regulations and accident reports, ensuring that decisions prioritize safety and compliance with regulations.
4. **Zero-Shot Planning**: Enable the vehicle to understand and handle unfamiliar scenarios without prior experience.
5. **Continuous Learning and Personalization**: Continuously learn to adapt to the driver's preferences and improve the driving experience over time.
6. **Transparency and Trust Building**: Allow LLMs to explain their decision-making process in simple language, thereby enhancing user trust in the technology.

Through these efforts, researchers hope to develop a human-centered autonomous driving ecosystem where LLMs act as the "brain," while perception modules, positioning systems, and other onboard devices serve as the "eyes" and "hands," working together to accomplish complex driving tasks.