Abstract:In this work, we propose an LLM-based BT generation framework to leverage the strengths of both for sequential manipulation planning. To enable human-robot collaborative task planning and enhance intuitive robot programming by nonexperts, the framework takes human instructions to initiate the generation of action sequences and human feedback to refine BT generation in runtime. All presented methods within the framework are tested on a real robotic assembly example, which uses a gear set model from the Siemens Robot Assembly Challenge. We use a single manipulator with a tool-changing mechanism, a common practice in flexible manufacturing, to facilitate robust grasping of a large variety of objects. Experimental results are evaluated regarding success rate, logical coherence, executability, time consumption, and token consumption. To our knowledge, this is the first human-guided LLM-based BT generation framework that unifies various plausible ways of using LLMs to fully generate BTs that are executable on the real testbed and take into account granular knowledge of tool use.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to automatically generate Behavior Trees (BTs) for sequential task planning in robotic operations. Specifically, the paper focuses on the following aspects: 1. **Automated Generation of Behavior Trees**: Traditional methods such as Finite State Machines (FSMs) are effective, but manually programming behavior trees still requires a significant amount of human effort and is time-consuming. The paper proposes a framework based on Large Language Models (LLMs) aimed at automating the generation of behavior trees. 2. **Cross-Domain Transfer and Replanning in Dynamic Environments**: Existing methods still face challenges in cross-domain transfer and replanning in different tasks and dynamic environments. The paper explores how to leverage LLMs to improve performance in these areas. 3. **Intuitive Human-Robot Interaction**: Achieving intuitive and straightforward Human-Robot Interaction (HRI) and collaboration in behavior tree-based robotic control systems remains an open problem. The paper proposes methods to guide behavior tree generation through human instructions and feedback to enhance the efficiency and accuracy of human-robot interaction. In summary, the main goal of the paper is to develop a behavior tree generation framework based on LLMs to achieve efficient automated task planning, and to optimize the generation process through human instructions and feedback, thereby enhancing the autonomy and adaptability of robots in complex tasks.

Behavior Tree Generation using Large Language Models for Sequential Manipulation Planning with Human Instructions and Feedback

LLM as BT-Planner: Leveraging LLMs for Behavior Tree Generation in Robot Task Planning

Integrating Intent Understanding and Optimal Behavior Planning for Behavior Tree Generation from Human Instructions

LLM-BRAIn: AI-driven Fast Generation of Robot Behaviour Tree based on Large Language Model

HBTP: Heuristic Behavior Tree Planning with Large Language Model Reasoning

LLM-BT: Performing Robotic Adaptive Tasks based on Large Language Models and Behavior Trees

A Study on Training and Developing Large Language Models for Behavior Tree Generation

Automatic Behavior Tree Expansion with LLMs for Robotic Manipulation

Generalizable Long-Horizon Manipulations with Large Language Models

BT Expansion: a Sound and Complete Algorithm for Behavior Planning of Intelligent Robots with Behavior Trees

Large Language Model-Based Code Generation for the Control of Construction Assembly Robots: A Hierarchical Generation Approach

Decision-Making in Robotic Grasping with Large Language Models.

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

BTGenBot: Behavior Tree Generation for Robotic Tasks with Lightweight LLMs

Task and Motion Planning with Large Language Models for Object Rearrangement

Human Planning of Robot Actions through LLM-guided State Machine Synthesis

Large Language Models for Orchestrating Bimanual Robots

Consolidating Trees of Robotic Plans Generated Using Large Language Models to Improve Reliability

Empowering Large Language Models on Robotic Manipulation with Affordance Prompting

Grounding Language Models in Autonomous Loco-manipulation Tasks

Non-Prehensile Tool-Object Manipulation by Integrating LLM-Based Planning and Manoeuvrability-Driven Controls