Abstract:The fusion of human-centric design and artificial intelligence capabilities has opened up new possibilities for next-generation autonomous vehicles that go beyond traditional transportation. These vehicles can dynamically interact with passengers and adapt to their preferences. This article proposes a novel framework that leverages large language models (LLMs) to enhance the decision-making process in autonomous vehicles. By utilizing LLMs’ contextual understanding abilities with specialized tools, we aim to integrate the language and reasoning capabilities of LLMs into autonomous vehicles. Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks, to explore LLMs’ interpretation, interaction, and reasoning in various scenarios. We also examine some well-defined real-time personalized driving tasks, demonstrating how LLMs can influence driving behaviors based on drivers’ verbal commands. Our empirical results highlight the substantial advantages of utilizing chain-of-thought prompting, leading to improved driving decisions and showing the potential for LLMs to enhance personalized driving experiences through ongoing verbal feedback. The proposed framework aims to transform autonomous vehicle operations, offering personalized support, transparent decision making, and continuous learning to enhance safety and effectiveness. We achieve user-centric, transparent, and adaptive autonomous driving ecosystems supported by the integration of LLMs into autonomous vehicles. Experiment videos are available at https://youtube.com/playlist?list=PLgcRcf9w8BmLJi_fqTGq-7KCZsbpEIE4a&si=dhH9lgaeSmB5K94t.

Personalized Autonomous Driving with Large Language Models: Field Experiments

Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles

Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles

Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles

Large Language Models for Autonomous Driving (LLM4AD): Concept, Benchmark, Simulation, and Real-Vehicle Experiment

Drive Like a Human: Rethinking Autonomous Driving with Large Language Models

LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

LMDrive: Closed-Loop End-to-End Driving with Large Language Models

LLM4Drive: A Survey of Large Language Models for Autonomous Driving

DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving

Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving

Facilitating Autonomous Driving Tasks with Large Language Models

Empowering Autonomous Driving with Large Language Models: A Safety Perspective

A Language Agent for Autonomous Driving

On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation

Driving Everywhere with Large Language Model Policy Adaptation

DriveLLM: Charting the Path Toward Full Autonomous Driving with Large Language Models

SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Data

DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model

Evaluation of Large Language Models for Decision Making in Autonomous Driving

Human-Centric Autonomous Systems with LLMs for User Command Reasoning