CARTIER: Cartographic lAnguage Reasoning Targeted at Instruction Execution for Robots

Dmitriy Rivkin,Nikhil Kakodkar,Francois Hogan,Bobak H. Baghi,Gregory Dudek

2024-02-02

Abstract:This work explores the capacity of large language models (LLMs) to address problems at the intersection of spatial planning and natural language interfaces for navigation. We focus on following complex instructions that are more akin to natural conversation than traditional explicit procedural directives typically seen in robotics. Unlike most prior work where navigation directives are provided as simple imperative commands (e.g., "go to the fridge"), we examine implicit directives obtained through conversational interactions.We leverage the 3D simulator AI2Thor to create household query scenarios at scale, and augment it by adding complex language queries for 40 object types. We demonstrate that a robot using our method CARTIER (Cartographic lAnguage Reasoning Targeted at Instruction Execution for Robots) can parse descriptive language queries up to 42% more reliably than existing LLM-enabled methods by exploiting the ability of LLMs to interpret the user interaction in the context of the objects in the scenario.

Robotics,Computation and Language

What problem does this paper attempt to address?

The paper aims to address the problem of enabling robots to understand and execute complex, natural language instructions, particularly those with implicit directives that appear in everyday conversations. Unlike traditional simple commands (such as "go to the fridge"), the paper focuses on more complex and conversational instructions, such as those expressed through multi-turn dialogues or descriptive sentences.

To solve this problem, the authors developed a method called CARTIER (Cartographic lAnguage Reasoning Targeted at Instruction Execution for Robots). This method leverages the capabilities of large language models (LLMs) to parse and understand these complex language queries and convert them into specific navigation instructions. Specifically, CARTIER first uses a pre-trained object detector to identify objects in the environment, then combines this information with the user's query, using LLMs to infer the user's true intent and determine the specific location the robot should go to.

To validate the effectiveness of CARTIER, the authors created a series of scenarios in an extended AI2Thor simulation environment, which included 40 different types of objects and 3 different types of queries (explicit, implicit, and conversational). The experimental results showed that CARTIER performed excellently in handling complex queries, especially in dealing with conversational queries, demonstrating a significant advantage over other methods.

Additionally, the paper introduces a real-world deployment example, showcasing that CARTIER can successfully navigate to the target location based on conversational instructions from users in a real-world environment. This indicates that CARTIER can effectively enhance a robot's ability to understand and execute natural language instructions, thereby improving the quality of human-robot interaction.