HumanPlus: Humanoid Shadowing and Imitation from Humans

Zipeng Fu,Qingqing Zhao,Qi Wu,Gordon Wetzstein,Chelsea Finn

2024-06-15

Abstract:One of the key arguments for building robots that have similar form factors to human beings is that we can leverage the massive human data for training. Yet, doing so has remained challenging in practice due to the complexities in humanoid perception and control, lingering physical gaps between humanoids and humans in morphologies and actuation, and lack of a data pipeline for humanoids to learn autonomous skills from egocentric vision. In this paper, we introduce a full-stack system for humanoids to learn motion and autonomous skills from human data. We first train a low-level policy in simulation via reinforcement learning using existing 40-hour human motion datasets. This policy transfers to the real world and allows humanoid robots to follow human body and hand motion in real time using only a RGB camera, i.e. shadowing. Through shadowing, human operators can teleoperate humanoids to collect whole-body data for learning different tasks in the real world. Using the data collected, we then perform supervised behavior cloning to train skill policies using egocentric vision, allowing humanoids to complete different tasks autonomously by imitating human skills. We demonstrate the system on our customized 33-DoF 180cm humanoid, autonomously completing tasks such as wearing a shoe to stand up and walk, unloading objects from warehouse racks, folding a sweatshirt, rearranging objects, typing, and greeting another robot with 60-100% success rates using up to 40 demonstrations. Project website: <a class="link-external link-https" href="https://humanoid-ai.github.io/" rel="external noopener nofollow">this https URL</a>

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning,Systems and Control

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper primarily addresses the following issues:

1. **Training Humanoid Robots from Human Data**:
- Utilizing large-scale human motion and skill data to train humanoid robots, enabling them to mimic human actions and autonomously complete tasks.
- Existing methods face numerous challenges in practical applications, including complex perception and control issues, physical differences between humans and humanoid robots, and the lack of effective data pipelines.

2. **Real-time Shadowing System**:
- Developed a real-time shadowing system based on a single RGB camera, allowing human operators to control the full-body movements of humanoid robots in real-time.
- This system achieves efficient full-body kinesthetic operation by estimating human movements in real-time and mapping them onto the humanoid robot.

3. **Efficient Imitation Learning Algorithm**:
- Proposed an imitation learning-based method that can quickly learn complex tasks from a small number of demonstrations (up to 40), such as standing and walking in shoes, unloading, folding clothes, etc.
- By combining visual input and high-degree-of-freedom control, the learning efficiency and robustness are improved.

### Main Contributions of the Paper

- **HumanPlus System**: A comprehensive humanoid robot system capable of learning complex autonomous skills from human data.
- **Real-time Shadowing System**: Achieved real-time full-body control using a single RGB camera and the Humanoid Shadowing Transformer.
- **Efficient Imitation Learning Algorithm**: Enabled efficient learning of complex skills from a small number of demonstrations through the Humanoid Imitation Transformer.