Abstract:The robotics community has consistently aimed to achieve generalizable robot manipulation with flexible natural language instructions. One of the primary challenges is that obtaining robot data fully annotated with both actions and texts is time-consuming and labor-intensive. However, partially annotated data, such as human activity videos without action labels and robot play data without language labels, is much easier to collect. Can we leverage these data to enhance the generalization capability of robots? In this paper, we propose GR-MG, a novel method which supports conditioning on both a language instruction and a goal image. During training, GR-MG samples goal images from trajectories and conditions on both the text and the goal image or solely on the image when text is unavailable. During inference, where only the text is provided, GR-MG generates the goal image via a diffusion-based image-editing model and condition on both the text and the generated image. This approach enables GR-MG to leverage large amounts of partially annotated data while still using language to flexibly specify tasks. To generate accurate goal images, we propose a novel progress-guided goal image generation model which injects task progress information into the generation process, significantly improving the fidelity and the performance. In simulation experiments, GR-MG improves the average number of tasks completed in a row of 5 from 3.35 to 4.04. In real-robot experiments, GR-MG is able to perform 47 different tasks and improves the success rate from 62.5% to 75.0% and 42.4% to 57.6% in simple and generalization settings, respectively. Code and checkpoints will be available at the project page: <a class="link-external link-https" href="https://gr-mg.github.io/" rel="external noopener nofollow">this https URL</a>.

GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields

Learning Generalizable Feature Fields for Mobile Manipulation

Learning Generalizable 3D Manipulation With 10 Demonstrations

Never-Ending Behavior-Cloning Agent for Robotic Manipulation

RPRG: Toward Real-time Robotic Perception, Reasoning and Grasping with One Multi-task Convolutional Neural Network.

RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation

Generalized Robot Learning Framework

ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation

Integrating Historical Learning and Multi-View Attention with Hierarchical Feature Fusion for Robotic Manipulation

GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation

Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation using Large Language Models

Programmatically Grounded, Compositionally Generalizable Robotic Manipulation

RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking

The Art of Imitation: Learning Long-Horizon Manipulation Tasks from Few Demonstrations

Transferring Foundation Models for Generalizable Robotic Manipulation

A Knowledge-Based Task Planning Approach for Robot Multi-Task Manipulation

Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration

Multi-Robot Collaborative Perception With Graph Neural Networks

GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal Conditioned Policy