Abstract:For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary, e.g. "can you get me the pink stuffed whale?" to their sensory observations and actions. This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information. We would like a robot's policy to be able to perceive and pick up the pink stuffed whale, even if it has never seen any data interacting with a stuffed whale before. Fortunately, static data on the internet has vast semantic information, and this information is captured in pre-trained vision-language models. In this paper, we study whether we can interface robot policies with these pre-trained models, with the aim of allowing robots to complete instructions involving object categories that the robot has never seen first-hand. We develop a simple approach, which we call Manipulation of Open-World Objects (MOO), which leverages a pre-trained vision-language model to extract object-identifying information from the language command and image, and conditions the robot policy on the current image, the instruction, and the extracted object information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments. In addition, we show how MOO generalizes to other, non-language-based input modalities to specify the object of interest such as finger pointing, and how it can be further extended to enable open-world navigation and manipulation. The project's website and evaluation videos can be found at <a class="link-external link-https" href="https://robot-moo.github.io/" rel="external noopener nofollow">this https URL</a>

Vision-Based Categorical Object Pose Estimation and Manipulation.

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

Calibration-Free Monocular Vision-Based Robot Manipulations With Occlusion Awareness

A Comprehensive Study of 3-D Vision-Based Robot Manipulation

RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation

Object Classification and Grasp Planning Using Visual and Tactile Sensing

A Robotic Semantic Grasping Method for Pick-and-place Tasks

Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructions

Open-World Object Manipulation using Pre-trained Vision-Language Models

Learning Robotic Manipulation through Visual Planning and Acting

Safe and Efficient Robot Manipulation: Task-Oriented Environment Modeling and Object Pose Estimation

Vision-based Robot Manipulation Learning via Human Demonstrations

A Vision-based Robot Grasping System

You Only Look at One: Category-Level Object Representations for Pose Estimation From a Single Example

Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraints

Vision-based Manipulation of Deformable and Rigid Objects Using Subspace Projections of 2D Contours

Research on object recognition and manipulator grasping strategy based on binocular vision

Nothing But Geometric Constraints: A Model-Free Method for Articulated Object Pose Estimation

A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter

Soft Contact Simulation and Manipulation Learning of Deformable Objects with Vision-based Tactile Sensor

Vision-based Manipulation from Single Human Video with Open-World Object Graphs