Abstract:Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the humanoid model and real humans. That is, motions of real humans may not be physically possible for the humanoid model. To overcome the dynamics mismatch, we propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space. During training, the RFC-based policy learns to apply residual forces to the humanoid to compensate for the dynamics mismatch and better imitate the reference motion. Experiments on a wide range of dynamic motions demonstrate that our approach outperforms state-of-the-art methods in terms of convergence speed and the quality of learned motions. Notably, we showcase a physics-based virtual character empowered by RFC that can perform highly agile ballet dance moves such as pirouette, arabesque and jeté. Furthermore, we propose a dual-policy control framework, where a kinematic policy and an RFC-based policy work in tandem to synthesize multi-modal infinite-horizon human motions without any task guidance or user input. Our approach is the first humanoid control method that successfully learns from a large-scale human motion dataset (Human3.6M) and generates diverse long-term motions. Code and videos are available at <a class="link-external link-https" href="https://www.ye-yuan.com/rfc" rel="external noopener nofollow">this https URL</a>.

Emergence of Human-comparable Balancing Behaviors by Deep Reinforcement Learning

Learning Whole-body Motor Skills for Humanoids

On the Emergence of Whole-body Strategies from Humanoid Robot Push-recovery Learning

Hybrid Autonomous Controller for Bipedal Robot Balance with Deep Reinforcement Learning and Pattern Generators

Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control

Autonomous Boundary of Human-Machine Collaboration System Based on Reinforcement Learning

Agile and versatile bipedal robot tracking control through reinforcement learning

Realizing full-body control of humanoid robots

Learning Ball-balancing Robot Through Deep Reinforcement Learning

Humanoid Balancing Behavior Featured by Underactuated Foot Motion

Achieving Stable High-Speed Locomotion for Humanoid Robots with Deep Reinforcement Learning

Learning of Long-Horizon Sparse-Reward Robotic Manipulator Tasks With Base Controllers

Learning Bipedal Walking for Humanoids with Current Feedback

A Heuristics-Based Reinforcement Learning Method to Control Bipedal Robots

Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante

A Biped Robot Learning to Walk like Human by Reinforcement Learning.

Real-World Humanoid Locomotion with Reinforcement Learning

Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models

Dynamic Bipedal Maneuvers through Sim-to-Real Reinforcement Learning

Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis