LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning

S P Sharan,Francesco Pittaluga,Vijay Kumar B G,Manmohan Chandraker

2023-12-30

Abstract:Although planning is a crucial component of the autonomous driving stack, researchers have yet to develop robust planning algorithms that are capable of safely handling the diverse range of possible driving scenarios. Learning-based planners suffer from overfitting and poor long-tail performance. On the other hand, rule-based planners generalize well, but might fail to handle scenarios that require complex driving maneuvers. To address these limitations, we investigate the possibility of leveraging the common-sense reasoning capabilities of Large Language Models (LLMs) such as GPT4 and Llama2 to generate plans for self-driving vehicles. In particular, we develop a novel hybrid planner that leverages a conventional rule-based planner in conjunction with an LLM-based planner. Guided by commonsense reasoning abilities of LLMs, our approach navigates complex scenarios which existing planners struggle with, produces well-reasoned outputs while also remaining grounded through working alongside the rule-based approach. Through extensive evaluation on the nuPlan benchmark, we achieve state-of-the-art performance, outperforming all existing pure learning- and rule-based methods across most metrics. Our code will be available at

Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to address the key challenges in the planning algorithms of autonomous vehicles, namely, how to develop robust planning algorithms that can safely handle various driving scenarios. Existing learning - based planners are prone to over - fitting and perform poorly in long - tailed performance, while rule - based planners, although having better generalization ability, may fail in scenarios requiring complex driving operations. To this end, the authors propose a new hybrid planning method that utilizes the common - sense reasoning capabilities of large - language models (LLMs) such as GPT4 and Llama2 to generate plans for autonomous vehicles. Specifically, they develop a novel hybrid planner that combines traditional rule - based planners with LLM - based planners. In this way, their method can navigate in complex scenarios that are difficult for existing planners to handle and maintain stability while maintaining logic by working in synergy with rule - based methods. Through extensive evaluation on the nuPlan benchmark, this method has achieved state - of - the - art performance on most metrics, surpassing all existing pure learning - based and rule - based methods.