Abstract:For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary, e.g. "can you get me the pink stuffed whale?" to their sensory observations and actions. This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information. We would like a robot's policy to be able to perceive and pick up the pink stuffed whale, even if it has never seen any data interacting with a stuffed whale before. Fortunately, static data on the internet has vast semantic information, and this information is captured in pre-trained vision-language models. In this paper, we study whether we can interface robot policies with these pre-trained models, with the aim of allowing robots to complete instructions involving object categories that the robot has never seen first-hand. We develop a simple approach, which we call Manipulation of Open-World Objects (MOO), which leverages a pre-trained vision-language model to extract object-identifying information from the language command and image, and conditions the robot policy on the current image, the instruction, and the extracted object information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments. In addition, we show how MOO generalizes to other, non-language-based input modalities to specify the object of interest such as finger pointing, and how it can be further extended to enable open-world navigation and manipulation. The project's website and evaluation videos can be found at <a class="link-external link-https" href="https://robot-moo.github.io/" rel="external noopener nofollow">this https URL</a>

MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

DeMoBot: Deformable Mobile Manipulation with Vision-based Sub-goal Retrieval

Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance

Manipulate-Anything: Automating Real-World Robots using Vision-Language Models

Open-World Object Manipulation using Pre-trained Vision-Language Models

Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination

Learning Multimodal Contact-Rich Skills from Demonstrations Without Reward Engineering

Dynamic Non-Prehensile Object Transport via Model-Predictive Reinforcement Learning

Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation

Vision-based Robot Manipulation Learning via Human Demonstrations

Extended Reality System for Robotic Learning from Human Demonstration

MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation

CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation

Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

Learning Task Planning from Multi-Modal Demonstration for Multi-Stage Contact-Rich Manipulation

Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation

Deep Dynamics Models for Learning Dexterous Manipulation

TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation

Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks