Abstract:Human motion synthesis is a long-standing problem with various applications in digital twins and the Metaverse. However, modern deep learning based motion synthesis approaches barely consider the physical plausibility of synthesized motions and consequently they usually produce unrealistic human motions. In order to solve this problem, we propose a system ``Skeleton2Humanoid'' which performs physics-oriented motion correction at test time by regularizing synthesized skeleton motions in a physics simulator. Concretely, our system consists of three sequential stages: (I) test time motion synthesis network adaptation, (II) skeleton to humanoid matching and (III) motion imitation based on reinforcement learning (RL). Stage I introduces a test time adaptation strategy, which improves the physical plausibility of synthesized human skeleton motions by optimizing skeleton joint locations. Stage II performs an analytical inverse kinematics strategy, which converts the optimized human skeleton motions to humanoid robot motions in a physics simulator, then the converted humanoid robot motions can be served as reference motions for the RL policy to imitate. Stage III introduces a curriculum residual force control policy, which drives the humanoid robot to mimic complex converted reference motions in accordance with the physical law. We verify our system on a typical human motion synthesis task, motion-in-betweening. Experiments on the challenging LaFAN1 dataset show our system can outperform prior methods significantly in terms of both physical plausibility and accuracy. Code will be released for research purposes at: <a class="link-external link-https" href="https://github.com/michaelliyunhao/Skeleton2Humanoid" rel="external noopener nofollow">this https URL</a>

Hierarchical Model-Based Imitation Learning for In-between Motion Synthesis

Imitation Learning of Hierarchical Driving Model: from Continuous Intention to Continuous Trajectory

Learning a Deep Motion Interpolation Network for Human Skeleton Animations

A GIM-based approach for biomimetic robot motion learning.

Motion Control of Bionic Robots Via Biomimetic Learning

Bi-Level Motion Imitation for Humanoid Robots

Hierarchical Motion Understanding via Motion Programs

Learning Hierarchical Non-Parametric Hidden Markov Model of Human Motion

Motion Optimization Based on Hierarchical Iterative Parameter Learning for Complicated Trajectory

Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening

Example-Based Automatic Music-Driven Conventional Dance Motion Synthesis

Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control

Robust Diffusion‐based Motion In‐betweening

HumanMimic: Learning Natural Locomotion and Transitions for Humanoid Robot via Wasserstein Adversarial Imitation

Learning Responsive Humanoid Motion Skills from Graph-Powered Motion Matching

Example-based Motion Synthesis Via Generative Motion Matching

Learning and Inferring Movement with Deep Generative Model

Competitive Motion Synthesis Based on Hybrid Control.

Accelerating Human Motion Imitation with Interactive Reinforcement Learning

Behavior Imitation for Manipulator Control and Grasping with Deep Reinforcement Learning

Motion In-Betweening with Phase Manifolds