Abstract:Our goal is to develop models that allow a robot to efficiently understand or “ground” natural language instructions in the context of its world representation. Contemporary approaches estimate correspondences between language instructions and possible groundings such as objects, regions, and goals for actions that the robot should execute. However, these approaches typically reason in relatively small domains and do not model abstract spatial concepts such as as “rows,” “columns,” or “groups” of objects and, hence, are unable to interpret an instruction such as “pick up the middle block in the row of five blocks.” In this paper, we introduce two new models for efficient natural language understanding of robot instructions. The first model, which we call the adaptive distributed correspondence graph (ADCG), is a probabilistic model for interpreting abstract concepts that require hierarchical reasoning over constituent concrete entities as well as notions of cardinality and ordinality. Abstract grounding variables form a Markov boundary over concrete groundings, effectively de-correlating them from the remaining variables in the graph. This structure reduces the complexity of model training and inference. Inference in the model is posed as an approximate search procedure that orders factor computation such that the estimated probable concrete groundings focus the search for abstract concepts towards likely hypothesis, pruning away improbable portions of the exponentially large space of abstractions. Further, we address the issue of scalability to complex domains and introduce a hierarchical extension to a second model termed the hierarchical adaptive distributed correspondence graph (HADCG). The model utilizes the abstractions in the ADCG but infers a coarse symbolic structure from the utterance and the environment model and then performs fine-grained inference over the reduced graphical model, further improving the efficiency of inference. Empirical evaluation demonstrates accurate grounding of abstract concepts embedded in complex natural language instructions commanding a robotic torso and a mobile robot. Further, the proposed approximate inference method allows significant efficiency gains compared with the baseline, with minimal trade-off in accuracy.

Online Grounding of Symbolic Planning Domains in Unknown Environments

Robot Active Neural Sensing and Planning in Unknown Cluttered Environments

Language Understanding for Field and Service Robots in a Priori Unknown Environments

Air-Ground Collaboration with SPOMP: Semantic Panoramic Online Mapping and Planning

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

Egocentric Planning for Scalable Embodied Task Achievement

Cognitive Planning for Object Goal Navigation using Generative AI Models

Portable Planner for Enhancing Ground Robots Exploration Performance in Unstructured Environments

Semantic Grounding for Long-Term Autonomy of Mobile Robots Towards Dynamic Object Search in Home Environments

Real-time map building and navigation for autonomous robots in unknown environments

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Task Planning

Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms

From Natural Language to Long-Range Path Plans in Outdoor Environments

Semantic Grounding for Long-Term Autonomy of Mobile Robots Toward Dynamic Object Search in Home Environments

EMPOWER: Embodied Multi-role Open-vocabulary Planning with Online Grounding and Execution

Grounding Robot Policies with Visuomotor Language Guidance

Work in Progress -- Automated Generation of Robotic Planning Domains from Observations

Providers-Clients-Robots: Framework for spatial-semantic planning for shared understanding in human-robot interaction

OrionNav: Online Planning for Robot Autonomy with Context-Aware LLM and Open-Vocabulary Semantic Scene Graphs

Robot Navigation in Unseen Spaces using an Abstract Map