ReplanVLM: Replanning Robotic Tasks with Visual Language Models

Aoran Mei,Guo-Niu Zhu,Huaxiang Zhang,Zhongxue Gan
2024-08-01
Abstract:Large language models (LLMs) have gained increasing popularity in robotic task planning due to their exceptional abilities in text analytics and generation, as well as their broad knowledge of the world. However, they fall short in decoding visual cues. LLMs have limited direct perception of the world, which leads to a deficient grasp of the current state of the world. By contrast, the emergence of visual language models (VLMs) fills this gap by integrating visual perception modules, which can enhance the autonomy of robotic task planning. Despite these advancements, VLMs still face challenges, such as the potential for task execution errors, even when provided with accurate instructions. To address such issues, this paper proposes a ReplanVLM framework for robotic task planning. In this study, we focus on error correction interventions. An internal error correction mechanism and an external error correction mechanism are presented to correct errors under corresponding phases. A replan strategy is developed to replan tasks or correct error codes when task execution fails. Experimental results on real robots and in simulation environments have demonstrated the superiority of the proposed framework, with higher success rates and robust error correction capabilities in open-world tasks. Videos of our experiments are available at <a class="link-external link-https" href="https://youtu.be/NPk2pWKazJc" rel="external noopener nofollow">this https URL</a>.
Robotics
What problem does this paper attempt to address?
The paper aims to address the limitations of large language models (LLMs) in robotic task planning, particularly their inadequacies in decoding visual cues. Although LLMs excel in text analysis and generation and possess extensive world knowledge, they have limited direct perception of the physical world, leading to insufficient understanding of the current world state. In contrast, visual language models (VLMs) compensate for this shortcoming by integrating visual perception modules, thereby enhancing the autonomy of robotic task planning. However, even with accurate instructions, VLMs still face challenges such as execution errors. To tackle these issues, the paper proposes a new framework called ReplanVLM, which focuses on error correction interventions. This framework includes internal and external error correction mechanisms that can rectify errors at the corresponding stages. When task execution fails, the framework replans the task or corrects the erroneous code. Experimental results show that this framework demonstrates superiority in both real robots and simulated environments, with higher success rates and robust error correction capabilities. In summary, the main objective of the paper is to enhance the capability of robotic task planning by introducing the ReplanVLM framework based on VLMs, particularly improving its adaptability and robustness in open-world tasks.