Abstract:The cooperative driving technology of Connected and Autonomous Vehicles (CAVs) is crucial for improving the efficiency and safety of transportation systems. Learning-based methods, such as Multi-Agent Reinforcement Learning (MARL), have demonstrated strong capabilities in cooperative decision-making tasks. However, existing MARL approaches still face challenges in terms of learning efficiency and performance. In recent years, Large Language Models (LLMs) have rapidly advanced and shown remarkable abilities in various sequential decision-making tasks. To enhance the learning capabilities of cooperative agents while ensuring decision-making efficiency and cost-effectiveness, we propose LDPD, a language-driven policy distillation method for guiding MARL exploration. In this framework, a teacher agent based on LLM trains smaller student agents to achieve cooperative decision-making through its own decision-making demonstrations. The teacher agent enhances the observation information of CAVs and utilizes LLMs to perform complex cooperative decision-making reasoning, which also leverages carefully designed decision-making tools to achieve expert-level decisions, providing high-quality teaching experiences. The student agent then refines the teacher's prior knowledge into its own model through gradient policy updates. The experiments demonstrate that the students can rapidly improve their capabilities with minimal guidance from the teacher and eventually surpass the teacher's performance. Extensive experiments show that our approach demonstrates better performance and learning efficiency compared to baseline methods.

What problem does this paper attempt to address?

### Problems the paper attempts to solve This paper aims to address the challenges faced by the cooperative driving techniques of connected and autonomous vehicles (CAVs) in multi - agent reinforcement learning (MARL). Specifically, existing MARL methods still have deficiencies in learning efficiency and performance. Although large language models (LLMs) perform well in various sequential decision - making tasks, in real - time dynamic traffic scenarios, existing LLMs have difficulty enabling CAVs to make decisions quickly and efficiently, and the high deployment cost limits their large - scale application. To meet these challenges, the authors propose a new method - language - driven policy distillation (LDPD), which combines the inference efficiency of MARL agents and the extensive world knowledge of LLMs. The LDPD framework guides student agents (based on MARL) by teacher agents (based on LLM) for efficient learning and exploration, thereby improving the efficiency and performance of cooperative decision - making. ### Main contributions 1. **Modeling the cooperative decision - making problem**: Model the cooperative decision - making problem of multiple CAVs as a multi - agent reinforcement learning (MARL) problem and use LLM to guide the exploration of CAVs. 2. **Proposing the LDPD framework**: Design a language - driven policy distillation framework to promote the rapid learning and efficient exploration of small - scale policy networks. 3. **Significant performance advantages**: Guided by the teacher agent, the performance of the student agent is significantly better than the baseline method, with obvious improvements in both learning efficiency and overall performance. ### Experimental verification The experimental results show that the LDPD method performs well in scenarios with different traffic densities and complexities. In particular, in the early stage of training, the exploration efficiency is significantly improved, effectively enhancing the learning efficiency of the student agent. Compared with the baseline method, LDPD shows better performance on multiple indicators. ### Formula presentation 1. **Reward function**: \[ r_{i,t}=\omega_c r_c+\omega_s r_s+\omega_h r_h+\omega_m r_m \] where \(\omega_c, \omega_s, \omega_h, \omega_m\) are positive weight coefficients, corresponding to the collision reward \(r_c\), the stable - speed reward \(r_s\), the headway - cost reward \(r_h\) and the merge - cost reward \(r_m\) respectively. 2. **Priority score**: \[ p_i = \alpha_1 p_m+\alpha_2 p_e+\alpha_3 p_h+\sigma_i \] where \(p_i\) represents the priority of agent \(i\), \(\alpha_1, \alpha_2, \alpha_3\) are positive weight parameters, corresponding to the merge indicator \(p_m\), the merge - end indicator \(p_e\) and the time - interval indicator \(p_h\) respectively. The random variable \(\sigma_i\sim N(0, 0.001)\) is used to avoid the problem of the same priority value. 3. **Merge indicator**: \[ p_m=\begin{cases} 0.5, & \text{if on the merge lane}\\ 0, & \text{otherwise} \end{cases} \] 4. **Merge - end priority**: \[ p_e=\begin{cases} \frac{x}{L}, & \text{if on the merge lane}\\ 0, & \text{otherwise} \end{cases} \] where \(L\) is the total length of the merge lane and \(x\) is the position of the vehicle on the ramp. 5. **Time - interval priority**: \[ p_h =-\log\left(\frac{d_{\text{headway}}}{v_t}\right) \] where \(d_{\text{headway}}\) is the current headway distance and \(v

Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learning

Learning to Cooperate: Application of Deep Reinforcement Learning for Online AGV Path Finding.

Towards Interactive and Learnable Cooperative Driving Automation: a Large Language Model-Driven Decision-Making Framework

LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learning

Communication-Efficient Decentralized Multi-Agent Reinforcement Learning for Cooperative Adaptive Cruise Control

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents

Multi-Agent Reinforcement Learning-Based Decision Making for Twin-Vehicles Cooperative Driving in Stochastic Dynamic Highway Environments

Multi-Vehicle Cooperative Decision-Making in Merging Area Based on Deep Multi-Agent Reinforcement Learning

YOLO-MARL: You Only LLM Once for Multi-agent Reinforcement Learning

Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Driving

A Multi-Agent Reinforcement Learning Approach For Safe and Efficient Behavior Planning Of Connected Autonomous Vehicles

Knowledge Distillation from Language-Oriented to Emergent Communication for Multi-Agent Remote Control

Communication-Aware Reinforcement Learning for Cooperative Adaptive Cruise Control

Driving Everywhere with Large Language Model Policy Adaptation

Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs

Large-Scale Traffic Signal Control Using a Novel Multiagent Reinforcement Learning

A Language Agent for Autonomous Driving

Stable and Efficient Shapley Value-Based Reward Reallocation for Multi-Agent Reinforcement Learning of Autonomous Vehicles

A Multi-teacher Policy Distillation Framework for Enhancing Zero-Shot Generalization of Autonomous Driving Policies

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic