Abstract:For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary, e.g. "can you get me the pink stuffed whale?" to their sensory observations and actions. This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information. We would like a robot's policy to be able to perceive and pick up the pink stuffed whale, even if it has never seen any data interacting with a stuffed whale before. Fortunately, static data on the internet has vast semantic information, and this information is captured in pre-trained vision-language models. In this paper, we study whether we can interface robot policies with these pre-trained models, with the aim of allowing robots to complete instructions involving object categories that the robot has never seen first-hand. We develop a simple approach, which we call Manipulation of Open-World Objects (MOO), which leverages a pre-trained vision-language model to extract object-identifying information from the language command and image, and conditions the robot policy on the current image, the instruction, and the extracted object information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments. In addition, we show how MOO generalizes to other, non-language-based input modalities to specify the object of interest such as finger pointing, and how it can be further extended to enable open-world navigation and manipulation. The project's website and evaluation videos can be found at <a class="link-external link-https" href="https://robot-moo.github.io/" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to enable robots to perform operational tasks involving object categories that have never been seen before according to human instructions. Specifically, researchers are concerned with how to make robots understand and execute instructions containing new words, which may refer to object categories that the robots have not encountered in the training data. For example, if a robot is asked "Can you help me get that pink plush whale?" and it has never seen or interacted with any plush animals before, how should it complete this task? The paper proposes a method. By using pre - trained vision - language models (VLMs), robots can identify and operate these new object categories, even if these objects do not appear in the robot's direct experience. This method aims to overcome the limitations of traditional robot learning methods, which usually rely on a limited number of object categories for training and are difficult to generalize to new, unseen objects.

Open-World Object Manipulation using Pre-trained Vision-Language Models

MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting

Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Maps

Vision-Based Categorical Object Pose Estimation and Manipulation.

Manipulate-Anything: Automating Real-World Robots using Vision-Language Models

Vision-based Manipulation from Single Human Video with Open-World Object Graphs

Object-Centric Instruction Augmentation for Robotic Manipulation

Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models

Adaptive Mobile Manipulation for Articulated Objects In the Open World

OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge

ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation

ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models

NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation

Vision-Language Foundation Models as Effective Robot Imitators

Task-oriented Robotic Manipulation with Vision Language Models

Understanding Contexts Inside Robot and Human Manipulation Tasks through a Vision-Language Model and Ontology System in a Video Stream

Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation

Robobarista: Learning to Manipulate Novel Objects via Deep Multimodal Embedding