NARRATE: Versatile Language Architecture for Optimal Control in Robotics

Seif Ismail,Antonio Arbues,Ryan Cotterell,René Zurbrügg,Carmen Amo Alonso

2024-03-16

Abstract:The impressive capabilities of Large Language Models (LLMs) have led to various efforts to enable robots to be controlled through natural language instructions, opening exciting possibilities for human-robot interaction The goal is for the motor-control task to be performed accurately, efficiently and safely while also enjoying the flexibility imparted by LLMs to specify and adjust the task through natural language. In this work, we demonstrate how a careful layering of an LLM in combination with a Model Predictive Control (MPC) formulation allows for accurate and flexible robotic control via natural language while taking into consideration safety constraints. In particular, we rely on the LLM to effectively frame constraints and objective functions as mathematical expressions, which are later used in the motor-control module via MPC. The transparency of the optimization formulation allows for interpretability of the task and enables adjustments through human feedback. We demonstrate the validity of our method through extensive experiments on long-horizon reasoning, contact-rich, and multi-object interaction tasks. Our evaluations show that NARRATE outperforms current existing methods on these benchmarks and effectively transfers to the real world on two different embodiments. Videos, Code and Prompts at

Robotics

What problem does this paper attempt to address?

The problem this paper attempts to address is how to efficiently, accurately, and safely control robots to perform complex operational tasks through natural language instructions. Specifically, the researchers aim to develop a system architecture that can translate high-level natural language instructions into precise low-level actions, enabling robots to understand human language descriptions and autonomously complete tasks based on these descriptions, while ensuring the safety and accuracy of the tasks. The challenge lies in combining the flexibility of natural language with the precise control of robots, especially when dealing with complex tasks such as long-term planning and multi-object interactions.

The method proposed in the paper is called NARRATE (Natural Language Robot Control and Adaptive Task Execution Architecture), which achieves this goal by combining large language models (LLMs) with model predictive control (MPC)-based optimization controllers. The core contribution of NARRATE is the use of LLMs to generate free-form mathematical expressions that can be directly used in MPC to solve manipulation tasks specified by natural language. This approach not only overcomes the limitations of using predefined primitives but also does not require specific robot control data, thus ensuring task accuracy and safety while maintaining flexibility.

Additionally, the design of NARRATE considers system interpretability and human-robot interaction, allowing task adjustments through natural language feedback, thereby improving system adaptability and performance. The researchers validated the effectiveness of NARRATE through a series of experiments, including long-term reasoning tasks in simulated environments, contact-rich tasks, and multi-robot collaboration tasks, demonstrating its potential for real-world applications.