OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning

Tairan He,Zhengyi Luo,Xialin He,Wenli Xiao,Chong Zhang,Weinan Zhang,Kris Kitani,Changliu Liu,Guanya Shi

2024-06-13

Abstract:We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets.

Robotics,Computer Vision and Pattern Recognition,Machine Learning,Systems and Control

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper "OmniH2O: Universal and Flexible Whole-Body Teleoperation and Learning for Humanoid Robots" aims to address the following issues:

1. **Whole-body control of full-sized humanoid robots**: Existing research mostly focuses on partial control of humanoid robots (such as lower limbs or upper limbs), lacking studies on whole-body coordinated control. OmniH2O attempts to achieve whole-body control of full-sized humanoid robots through a universal control interface, including dexterous hand manipulation and stable walking.

2. **Efficient teleoperation and autonomy**: Traditional teleoperation systems require expensive equipment (such as motion capture systems and exoskeletons), limiting large-scale data collection. OmniH2O proposes a flexible teleoperation method based on virtual reality (VR) headsets, RGB cameras, and language instructions, and achieves autonomous control by learning from teleoperation data.

3. **Robust motion imitation strategy**: For deployment in the real world, a robust control strategy capable of handling complex environments and tasks is needed. OmniH2O uses a teacher-student distillation framework, training a high-performance teacher policy with privileged information and transferring its knowledge to a simplified student policy suitable for the real world.

4. **Generation and utilization of large-scale motion datasets**: Existing robot datasets mainly focus on fixed-base robotic arm platforms, lacking data for full-sized humanoid robots. OmniH2O collects a dataset of full-sized humanoid robots performing various daily tasks and demonstrates how to use this data to learn autonomous strategies.

### Main Contributions

1. **Proposed a universal control pipeline for full-sized humanoid robots**: This pipeline supports whole-body dexterous manipulation and autonomy, capable of being controlled through multiple input sources (such as VR, RGB cameras, language instructions, etc.).
2. **Validated the superior performance of large-scale motion tracking in simulation and the real world**: Experiments demonstrated the efficiency and robustness of OmniH2O in motion imitation tasks.
3. **Released the first whole-body control dataset for full-sized humanoid robots**: This dataset includes 6 daily tasks, showcasing methods for learning whole-body skills of humanoid robots from teleoperation datasets.

### Experimental Results

1. **Motion tracking performance**: OmniH2O exhibited excellent motion tracking capabilities in both simulation and the real world, especially in whole-body coordinated control.
2. **Diverse human control interfaces**: Flexible control of humanoid robots was achieved through various interfaces such as VR headsets, RGB cameras, and language instructions.
3. **Robustness testing**: OmniH2O demonstrated good stability under different terrains and interference conditions.
4. **Autonomous control**: By integrating GPT-4o and learning from teleoperation data, autonomous control of humanoid robots was achieved.

Overall, OmniH2O provides a comprehensive solution for whole-body control and autonomy of full-sized humanoid robots, showcasing its potential in practical applications.