Abstract:Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills. Towards this goal, recent works have integrated semantic representations from large-scale pretrained vision-language (VL) models into manipulation models, imparting them with more general reasoning capabilities. However, we show that the conventional pretraining-finetuning pipeline for integrating such representations entangles the learning of domain-specific action information and domain-general visual information, leading to less data-efficient training and poor generalization to unseen objects and tasks. To this end, we propose ProgramPort, a modular approach to better leverage pretrained VL models by exploiting the syntactic and semantic structures of language instructions. Our framework uses a semantic parser to recover an executable program, composed of functional modules grounded on vision and action across different modalities. Each functional module is realized as a combination of deterministic computation and learnable neural networks. Program execution produces parameters to general manipulation primitives for a robotic end-effector. The entire modular network can be trained with end-to-end imitation learning objectives. Experiments show that our model successfully disentangles action and perception, translating to improved zero-shot and compositional generalization in a variety of manipulation behaviors. Project webpage at: \url{<a class="link-external link-https" href="https://progport.github.io" rel="external noopener nofollow">this https URL</a>}.

What problem does this paper attempt to address?

The paper primarily aims to address the issue of robots needing to possess rich manipulation skills and the ability to semantically reason about when to apply these skills in real-world operations. To achieve this goal, recent works have attempted to integrate the semantic representations from large-scale pre-trained Vision-Language (VL) models into manipulation models to endow them with more general reasoning capabilities. However, the paper points out that the traditional pre-training-fine-tuning pipeline entangles domain-specific action information with domain-general visual information when integrating such representations, leading to inefficient training and poor generalization performance, especially when faced with unseen objects and tasks. To address the above issues, the authors propose a method called PROGRAMPORT. This is a modular approach that leverages pre-trained Vision-Language models (such as CLIP) and the syntactic and semantic structures in natural language instructions to better utilize these models. Specifically, the framework uses a semantic parser to recover executable programs composed of functional modules that are grounded in different modalities (vision and action). Each functional module is a combination of deterministic computation and learnable neural networks. Program execution generates parameters for general manipulation primitives for the robot's end-effector. The entire modular network can be trained through end-to-end imitation learning objectives. Experiments show that the model successfully separates action and perception, thereby achieving improved zero-shot and compositional generalization across various manipulation behaviors. In summary, the main contribution of the paper is the proposal of a modular framework that effectively utilizes pre-trained Vision-Language models, which not only improves data efficiency but also enhances the model's generalization ability to unseen objects and tasks.

Programmatically Grounded, Compositionally Generalizable Robotic Manipulation

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

Grounding Language for Robotic Manipulation via Skill Library

Learning Neuro-symbolic Programs for Language Guided Robot Manipulation

Transferring Foundation Models for Generalizable Robotic Manipulation

Learning Robotic Manipulation through Visual Planning and Acting

ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation

Robobarista: Learning to Manipulate Novel Objects via Deep Multimodal Embedding

Enhancing Interpretability and Interactivity in Robot Manipulation: A Neurosymbolic Approach

Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations

Development of Compositionality and Generalization through Interactive Learning of Language and Action of Robots

Learning Generalizable 3D Manipulation With 10 Demonstrations

Grounding Object Relations in Language-Conditioned Robotic Manipulation with Semantic-Spatial Reasoning

RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking

Manipulate-Anything: Automating Real-World Robots using Vision-Language Models

ManiFoundation Model for General-Purpose Robotic Manipulation of Contact Synthesis with Arbitrary Objects and Robots

Open-World Object Manipulation using Pre-trained Vision-Language Models

Human-oriented Representation Learning for Robotic Manipulation

Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots

Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models

Learning Latent Object-Centric Representations for Visual-Based Robot Manipulation