Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs

Yusuke Mikami,Andrew Melnik,Jun Miura,Ville Hautamäki

2024-04-06

Abstract:We demonstrate experimental results with LLMs that address robotics task planning problems. Recently, LLMs have been applied in robotics task planning, particularly using a code generation approach that converts complex high-level instructions into mid-level policy codes. In contrast, our approach acquires text descriptions of the task and scene objects, then formulates task planning through natural language reasoning, and outputs coordinate level control commands, thus reducing the necessity for intermediate representation code as policies with pre-defined APIs. Our approach is evaluated on a multi-modal prompt simulation benchmark, demonstrating that our prompt engineering experiments with natural language reasoning significantly enhance success rates compared to its absence. Furthermore, our approach illustrates the potential for natural language descriptions to transfer robotics skills from known tasks to previously unseen tasks. The project website:

Robotics,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

This paper attempts to address the problem of how to utilize large language models (LLMs) to directly generate coordinate-level control commands through natural language reasoning in robot task planning, rather than relying on code generation methods. Specifically:

1. **Conversion from High-Level Instructions to Low-Level Execution Commands**: The research aims to decompose human-level high-level instructions into specific action commands that robots can execute. Traditional methods are usually task-specific and struggle to achieve open vocabulary object detection, new task generalization, and reduced complexity during the training process.

2. **Natural Language Description and Robot Skill Transfer**: A new framework is proposed that can formulate task planning through natural language descriptions and output coordinate-level control commands that can be directly executed, thereby reducing the need for predefined APIs as strategies. This approach makes it possible to transfer robot skills from known tasks to unknown tasks.

3. **Improving Zero-Shot Adaptation Capability**: While existing LLM-based research emphasizes the application of task-level zero-shot scenarios in robot task planning, this paper further explores how to enhance the ability to handle complex new situations and new tasks by leveraging knowledge from previously encountered similar tasks.

In summary, this paper focuses on demonstrating a new approach different from existing code generation methods, namely robot task planning through natural language reasoning, to improve the performance and flexibility of robots when facing new tasks.