Opt2Skill: Imitating Dynamically-feasible Whole-Body Trajectories for Versatile Humanoid Loco-Manipulation

Fukang Liu,Zhaoyuan Gu,Yilin Cai,Ziyi Zhou,Shijie Zhao,Hyunyoung Jung,Sehoon Ha,Yue Chen,Danfei Xu,Ye Zhao

2024-10-29

Abstract:Humanoid robots are designed to perform diverse loco-manipulation tasks. However, they face challenges due to their high-dimensional and unstable dynamics, as well as the complex contact-rich nature of the tasks. Model-based optimal control methods offer precise and systematic control but are limited by high computational complexity and accurate contact sensing. On the other hand, reinforcement learning (RL) provides robustness and handles high-dimensional spaces but suffers from inefficient learning, unnatural motion, and sim-to-real gaps. To address these challenges, we introduce Opt2Skill, an end-to-end pipeline that combines model-based trajectory optimization with RL to achieve robust whole-body loco-manipulation. We generate reference motions for the Digit humanoid robot using differential dynamic programming (DDP) and train RL policies to track these trajectories. Our results demonstrate that Opt2Skill outperforms pure RL methods in both training efficiency and task performance, with optimal trajectories that account for torque limits enhancing trajectory tracking. We successfully transfer our approach to real-world applications.

Robotics

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address the challenges faced by humanoid robots when performing diverse motion and manipulation tasks. Specifically, the paper focuses on the following points:

1. **High-dimensional and unstable dynamics**: Humanoid robots possess high-dimensional and unstable dynamic characteristics, making control complex and difficult to achieve precise and stable motion.
2. **Complex contact-rich tasks**: Humanoid robots need to handle various contact-rich tasks, such as carrying heavy objects, climbing stairs, and performing agile skills (e.g., jumping). The complexity of these tasks further increases the difficulty of control.
3. **Limitations of model predictive control**: Model-based optimal control methods, while providing precise and systematic control, have high computational complexity and require accurate contact sensing.
4. **Limitations of reinforcement learning**: Reinforcement learning (RL) methods, although capable of handling high-dimensional spaces and providing robustness, have shortcomings in learning efficiency, natural motion, and the sim-to-real gap.

To address these challenges, the paper introduces **Opt2Skill**, an end-to-end pipeline that combines model-based trajectory optimization and reinforcement learning to achieve robust whole-body motion and manipulation for humanoid robots. By generating reference motions and training RL policies to track these trajectories, Opt2Skill outperforms pure RL methods in training efficiency and task performance and has been successfully applied in real-world scenarios.