Abstract:Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models. Unfortunately, applying such models to settings with embodied agents, such as robots, is challenging due to their lack of experience with the physical world, inability to parse non-language observations, and ignorance of rewards or safety constraints that robots may require. On the other hand, language-conditioned robotic policies that learn from interaction data can provide the necessary grounding that allows the agent to be correctly situated in the real world, but such policies are limited by the lack of high-level semantic understanding due to the limited breadth of the interaction data available for training them. Thus, if we want to make use of the semantic knowledge in a language model while still situating it in an embodied setting, we must construct an action sequence that is both likely according to the language model and also realizable according to grounded models of the environment. We frame this as a problem similar to probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under a set of grounded model objectives. We demonstrate how such grounded models can be obtained across three simulation and real-world domains, and that the proposed decoding strategy is able to solve complex, long-horizon embodiment tasks in a robotic setting by leveraging the knowledge of both models. The project's website can be found at <a class="link-external link-http" href="http://grounded-decoding.github.io" rel="external noopener nofollow">this http URL</a>.

Collaborative Language Grounding Toward Situated Human‐Robot Dialogue

Collaborative Effort Towards Common Ground in Situated Human-Robot Dialogue.

Grounding Language for Robotic Manipulation via Skill Library

Natural Language for Human-Robot Collaboration: Problems Beyond Language Grounding

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

Towards Understanding Language through Perception in Situated Human-Robot Interaction: From Word Grounding to Grammar Induction

Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions

Crossmodal Language Grounding in an Embodied Neurocognitive Model

Language-Grounded Control for Coordinated Robot Motion and Speech

Integrating Word Acquisition and Referential Grounding Towards Physical World Interaction

Human-Robot Dialogue Annotation for Multi-Modal Common Ground

Trust in Language Grounding: a new AI challenge for human-robot teams

Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication

Grounding Language with Visual Affordances over Unstructured Data

Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Units

Human Robot Collaboration: an Augmented Reality Approach—A Literature Review and Analysis

Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue

Modeling Collaborative Referring for Situated Referential Grounding

Conversational Language Models for Human-in-the-Loop Multi-Robot Coordination