Abstract:Our goal is to develop models that allow a robot to efficiently understand or “ground” natural language instructions in the context of its world representation. Contemporary approaches estimate correspondences between language instructions and possible groundings such as objects, regions, and goals for actions that the robot should execute. However, these approaches typically reason in relatively small domains and do not model abstract spatial concepts such as as “rows,” “columns,” or “groups” of objects and, hence, are unable to interpret an instruction such as “pick up the middle block in the row of five blocks.” In this paper, we introduce two new models for efficient natural language understanding of robot instructions. The first model, which we call the adaptive distributed correspondence graph (ADCG), is a probabilistic model for interpreting abstract concepts that require hierarchical reasoning over constituent concrete entities as well as notions of cardinality and ordinality. Abstract grounding variables form a Markov boundary over concrete groundings, effectively de-correlating them from the remaining variables in the graph. This structure reduces the complexity of model training and inference. Inference in the model is posed as an approximate search procedure that orders factor computation such that the estimated probable concrete groundings focus the search for abstract concepts towards likely hypothesis, pruning away improbable portions of the exponentially large space of abstractions. Further, we address the issue of scalability to complex domains and introduce a hierarchical extension to a second model termed the hierarchical adaptive distributed correspondence graph (HADCG). The model utilizes the abstractions in the ADCG but infers a coarse symbolic structure from the utterance and the environment model and then performs fine-grained inference over the reduced graphical model, further improving the efficiency of inference. Empirical evaluation demonstrates accurate grounding of abstract concepts embedded in complex natural language instructions commanding a robotic torso and a mobile robot. Further, the proposed approximate inference method allows significant efficiency gains compared with the baseline, with minimal trade-off in accuracy.

Integrating Word Acquisition and Referential Grounding Towards Physical World Interaction

Collaborative Language Grounding Toward Situated Human‐Robot Dialogue

Grounding Language for Robotic Manipulation via Skill Library

Modeling Collaborative Referring for Situated Referential Grounding

Context-based Word Acquisition for Situated Dialogue in a Virtual World

Towards Mediating Shared Perceptual Basis in Situated Dialogue.

Collaborative Effort Towards Common Ground in Situated Human-Robot Dialogue.

Probabilistic Labeling for Efficient Referential Grounding Based on Collaborative Discourse.

Interactive Grounded Language Acquisition and Generalization in a 2D World

Grounding Language in Multi-Perspective Referential Communication

Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction

Reference-Centric Models for Grounded Collaborative Dialogue

Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms

Towards Understanding Language through Perception in Situated Human-Robot Interaction: From Word Grounding to Grammar Induction

Grounded Lexicon Acquisition - Case Studies in Spatial Language

Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition

Towards Situated Dialogue: Revisiting Referring Expression Generation.

Generalized Grounding Graphs: A Probabilistic Framework for Understanding Grounded Commands

Crossmodal Language Grounding in an Embodied Neurocognitive Model

Natural Language for Human-Robot Collaboration: Problems Beyond Language Grounding

Understanding Early Word Learning in Situated Artificial Agents