Abstract:The fusion of human-centric design and artificial intelligence capabilities has opened up new possibilities for next-generation autonomous vehicles that go beyond traditional transportation. These vehicles can dynamically interact with passengers and adapt to their preferences. This article proposes a novel framework that leverages large language models (LLMs) to enhance the decision-making process in autonomous vehicles. By utilizing LLMs’ contextual understanding abilities with specialized tools, we aim to integrate the language and reasoning capabilities of LLMs into autonomous vehicles. Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks, to explore LLMs’ interpretation, interaction, and reasoning in various scenarios. We also examine some well-defined real-time personalized driving tasks, demonstrating how LLMs can influence driving behaviors based on drivers’ verbal commands. Our empirical results highlight the substantial advantages of utilizing chain-of-thought prompting, leading to improved driving decisions and showing the potential for LLMs to enhance personalized driving experiences through ongoing verbal feedback. The proposed framework aims to transform autonomous vehicle operations, offering personalized support, transparent decision making, and continuous learning to enhance safety and effectiveness. We achieve user-centric, transparent, and adaptive autonomous driving ecosystems supported by the integration of LLMs into autonomous vehicles. Experiment videos are available at https://youtube.com/playlist?list=PLgcRcf9w8BmLJi_fqTGq-7KCZsbpEIE4a&si=dhH9lgaeSmB5K94t.

UnstrPrompt: Large Language Model Prompt for Driving in Unstructured Scenarios

Language Prompt for Autonomous Driving

Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs

Prompting Frameworks for Large Language Models: A Survey

Towards Goal-oriented Prompt Engineering for Large Language Models: A Survey

Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles

Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles

Large Language Models are Good Multi-lingual Learners : When LLMs Meet Cross-lingual Prompts

PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Driving

Automatic Prompt Selection for Large Language Models

Personalized Autonomous Driving with Large Language Models: Field Experiments

Visual Prompting in Multimodal Large Language Models: A Survey

Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles

PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models

Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Driving

Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning

Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions

RePrompt: Planning by Automatic Prompt Engineering for Large Language Models Agents

LMDrive: Closed-Loop End-to-End Driving with Large Language Models

Large Language Models for Autonomous Driving (LLM4AD): Concept, Benchmark, Simulation, and Real-Vehicle Experiment

Prompt Space Optimizing Few-shot Reasoning Success with Large Language Models