Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans

Homanga Bharadhwaj,Abhinav Gupta,Vikash Kumar,Shubham Tulsiani

2023-12-02

Abstract:We pursue the goal of developing robots that can interact zero-shot with generic unseen objects via a diverse repertoire of manipulation skills and show how passive human videos can serve as a rich source of data for learning such generalist robots. Unlike typical robot learning approaches which directly learn how a robot should act from interaction data, we adopt a factorized approach that can leverage large-scale human videos to learn how a human would accomplish a desired task (a human plan), followed by translating this plan to the robots embodiment. Specifically, we learn a human plan predictor that, given a current image of a scene and a goal image, predicts the future hand and object configurations. We combine this with a translation module that learns a plan-conditioned robot manipulation policy, and allows following humans plans for generic manipulation tasks in a zero-shot manner with no deployment-time training. Importantly, while the plan predictor can leverage large-scale human videos for learning, the translation module only requires a small amount of in-domain data, and can generalize to tasks not seen during training. We show that our learned system can perform over 16 manipulation skills that generalize to 40 objects, encompassing 100 real-world tasks for table-top manipulation and diverse in-the-wild manipulation. <a class="link-external link-https" href="https://homangab.github.io/hopman/" rel="external noopener nofollow">this https URL</a>

Robotics,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The main goal of this paper is to develop a robotic system capable of performing various daily operational tasks in the real world with zero-shot learning. Specifically, the researchers aim for the robot to achieve this by learning human interaction plans and translating them into actual robotic operations.

### Main Contributions

1. **Proposed a Method**: Learning goal-conditioned hand-object interaction plan predictions from everyday life videos.
2. **Developed a Framework**: Viewing robotic operations as translations of predicted hand-object plans, thereby leveraging easily accessible human video data to learn diverse operational skills.
3. **Demonstrated the Overall Framework**: Conducted experiments on 100 tasks involving 40 objects and 16 skills, and evaluated the generalization ability in structured ways for both desktop operations and unknown scenarios.

### Method Overview

1. **Human Plan Prediction Model**: Using a diffusion model to predict future hand and object masks from diverse passive human videos.
2. **Robot Action Translation Model**: Translating the predicted hand-object plans into actual robotic action sequences.

### Experimental Results

The researchers demonstrated the generalization ability of their framework with diverse real-world objects in unseen scenarios. The experiments included both desktop and field operation settings, with detailed definitions of different levels of generalization standards. The results showed that the framework achieved significant success rates in both standard generalization (G) and strong generalization (SG), outperforming methods that rely solely on in-domain data or other baseline methods.