Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Driving

Sihao Wu,Jiaxu Liu,Xiangyu Yin,Guangliang Cheng,Xingyu Zhao,Meng Fang,Xinping Yi,Xiaowei Huang

2024-10-20

Abstract:The integration of Large Language Models (LLMs) into autonomous driving systems demonstrates strong common sense and reasoning abilities, effectively addressing the pitfalls of purely data-driven methods. Current LLM-based agents require lengthy inference times and face challenges in interacting with real-time autonomous driving environments. A key open question is whether we can effectively leverage the knowledge from LLMs to train an efficient and robust Reinforcement Learning (RL) agent. This paper introduces RAPID, a novel \underline{\textbf{R}}obust \underline{\textbf{A}}daptive \underline{\textbf{P}}olicy \underline{\textbf{I}}nfusion and \underline{\textbf{D}}istillation framework, which trains specialized mix-of-policy RL agents using data synthesized by an LLM-based driving agent and online adaptation. RAPID features three key designs: 1) utilization of offline data collected from an LLM agent to distil expert knowledge into RL policies for faster real-time inference; 2) introduction of robust distillation in RL to inherit both performance and robustness from LLM-based teacher; and 3) employment of a mix-of-policy approach for joint decision decoding with a policy adapter. Through fine-tuning via online environment interaction, RAPID reduces the forgetting of LLM knowledge while maintaining adaptability to different tasks. Extensive experiments demonstrate RAPID's capability to effectively integrate LLM knowledge into scaled-down RL policies in an efficient, adaptable, and robust way. Code and checkpoints will be made publicly available upon acceptance.

Robotics,Artificial Intelligence

What problem does this paper attempt to address?

The main problem this paper attempts to address is: how to develop an efficient, adaptive, and robust autonomous driving agent that can leverage the capabilities of large language models (LLMs). Specifically, the paper aims to overcome several challenges currently faced by LLMs in autonomous driving, including long inference times, difficulty in continuously collecting data and learning in real-time dynamic driving environments, and the susceptibility of small models to adversarial attacks.

To tackle these challenges, the authors propose the RAPID framework, which stands for "Robust Adaptive Policy Infusion and Distillation." RAPID achieves its goals through the following three key designs:
1. **Offline Data Utilization**: Using offline data collected from the LLM agent to distill expert knowledge into RL policies, thereby accelerating real-time inference speed.
2. **Robust Distillation**: Introducing robust distillation techniques to ensure that the RL policies inherit not only the performance but also the robustness of the LLM teacher.
3. **Multi-Policy Fusion**: Employing a multi-policy fusion approach for joint decision decoding and using policy adapters.

Through these designs, RAPID is able to reduce the forgetting of LLM knowledge while maintaining adaptability to different tasks. Experimental results show that RAPID can effectively integrate the knowledge of LLMs into a scaled-down RL policy, achieving an efficient, adaptive, and robust autonomous driving system.