Action Contextualization: Adaptive Task Planning and Action Tuning using Large Language Models

Sthithpragya Gupta,Kunpeng Yao,Loïc Niederhauser,Aude Billard

2024-07-29

Abstract:Large Language Models (LLMs) present a promising frontier in robotic task planning by leveraging extensive human knowledge. Nevertheless, the current literature often overlooks the critical aspects of robots' adaptability and error correction. This work aims to overcome this limitation by enabling robots to modify their motions and select the most suitable task plans based on the context. We introduce a novel framework to achieve action contextualization, aimed at tailoring robot actions to the context of specific tasks, thereby enhancing adaptability through applying LLM-derived contextual insights. Our framework integrates motion metrics that evaluate robot performances for each motion to resolve redundancy in planning. Moreover, it supports online feedback between the robot and the LLM, enabling immediate modifications to the task plans and corrections of errors. An overall success rate of 81.25% has been achieved through extensive experimental validation. Finally, when integrated with dynamical system (DS)-based robot controllers, the robotic arm-hand system demonstrates its proficiency in autonomously executing LLM-generated motion plans for sequential table-clearing tasks, rectifying errors without human intervention, and showcasing robustness against external disturbances. Our proposed framework also features the potential to be integrated with modular control approaches, significantly enhancing robots' adaptability and autonomy in performing sequential tasks in the real world.

Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use large - language models (LLMs) to improve the adaptability and error - correction ability of robots in robot task planning and action execution. Specifically, current research often ignores the adaptability and error - correction ability of robots when facing different task environments. This paper proposes a new framework, aiming to enhance these abilities by enabling robots to adjust their actions according to the specific context of the task and select the most appropriate task plan. This framework not only integrates motion indicators for evaluating robot performance, but also supports online feedback between the robot and the LLM, so as to immediately modify the task plan and correct errors. In addition, this framework also demonstrates the ability to integrate with robot controllers based on dynamic systems, significantly improving the autonomy and adaptability of robots when performing sequential tasks.