Abstract:Contemporary approaches to perception, planning, estimation, and control have allowed robots to operate robustly as our remote surrogates in uncertain, unstructured environments. This progress now creates an opportunity for robots to operate not only in isolation, but also with and alongside humans in our complex environments. Realizing this opportunity requires an efficient and flexible medium through which humans can communicate with collaborative robots. Natural language provides one such medium, and through significant progress in statistical methods for natural-language understanding, robots are now able to interpret a diverse array of free-form navigation, manipulation, and mobile-manipulation commands. However, most contemporary approaches require a detailed, prior spatial-semantic map of the robot’s environment that models the space of possible referents of an utterance. Consequently, these methods fail when robots are deployed in new, previously unknown, or partially-observed environments, particularly when mental models of the environment differ between the human operator and the robot. This paper provides a comprehensive description of a novel learning framework that allows field and service robots to interpret and correctly execute natural-language instructions in a priori unknown, unstructured environments. Integral to our approach is its use of language as a “sensor”—inferring spatial, topological, and semantic information implicit in natural-language utterances and then exploiting this information to learn a distribution over a latent environment model. We incorporate this distribution in a probabilistic, language grounding model and infer a distribution over a symbolic representation of the robot’s action space, consistent with the utterance. We use imitation learning to identify a belief-space policy that reasons over the environment and behavior distributions. We evaluate our framework through a variety of different navigation and mobile-manipulation experiments involving an unmanned ground vehicle, a robotic wheelchair, and a mobile manipulator, demonstrating that the algorithm can follow natural-language instructions without prior knowledge of the environment.

Prospection: Interpretable Plans From Language By Predicting the Future

Visual Robot Task Planning

Learning to Imagine Manipulation Goals for Robot Task Planning

Plan Explicability and Predictability for Robot Task Planning

From Natural Language to Long-Range Path Plans in Outdoor Environments

Generating Plans that Predict Themselves

Language Understanding for Field and Service Robots in a Priori Unknown Environments

Points2Plans: From Point Clouds to Long-Horizon Plans with Composable Relational Dynamics

CogExplore: Contextual Exploration with Language-Encoded Environment Representations

Inferring Robot Task Plans from Human Team Meetings: A Generative Modeling Approach with Logic-Based Prior

Work in Progress -- Automated Generation of Robotic Planning Domains from Observations

ProgPrompt: Generating Situated Robot Task Plans using Large Language Models

Plan Explainability and Predictability for Cobots

Cognitive Planning for Object Goal Navigation using Generative AI Models

ProgPrompt: program generation for situated robot task planning using large language models

One to rule them all: natural language to bind communication, perception and action

Learning Models for Following Natural Language Directions in Unknown Environments

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

RePLan: Robotic Replanning with Perception and Language Models

Autonomous Planning of Service Robot Based on Natural Language Tasks in Intelligent Space

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Task Planning