Abstract:We introduce a novel method for synthesizing dance motions that follow the emotions and contents of a piece of music. Our method employs a learning-based approach to model the music to motion mapping relationship embodied in example dance motions along with those motions' accompanying background music. A key step in our method is to train a music to motion matching quality rating function through learning the music to motion mapping relationship exhibited in synchronized music and dance motion data, which were captured from professional human dance performance. To generate an optimal sequence of dance motion segments to match with a piece of music, we introduce a constraint-based dynamic programming procedure. This procedure considers both music to motion matching quality and visual smoothness of a resultant dance motion sequence. We also introduce a two-way evaluation strategy, coupled with a GPU-based implementation, through which we can execute the dynamic programming process in parallel, resulting in significant speedup. To evaluate the effectiveness of our method, we quantitatively compare the dance motions synthesized by our method with motion synthesis results by several peer methods using the motions captured from professional human dancers' performance as the gold standard. We also conducted several medium-scale user studies to explore how perceptually our dance motion synthesis method can outperform existing methods in synthesizing dance motions to match with a piece of music. These user studies produced very positive results on our music-driven dance motion synthesis experiments for several Asian dance genres, confirming the advantages of our method.

The SOMN-HMM Model and Its Application to Automatic Synthesis of 3D Character Animations

Learning Style-Directed Dynamics Of Human Motion For Automatic Motion Synthesis

Real-Time Synthesis of 3d Animations by Learning Parametric Gaussians Using Self-Organizing Mixture Networks

Learning Hierarchical Non-Parametric Hidden Markov Model of Human Motion

Learning a Deep Motion Interpolation Network for Human Skeleton Animations

Supervised Learning of Motion Style for Real-time Synthesis of 3D Character Animations

3D Human Motion Synthesis Based on Nonlinear Manifold Learning

Key-styling: learning motion style for real-time synthesis of 3D animation: Research Articles

Key-styling: learning motion style for real-time synthesis of 3D animation

A Human Motion Feature Based on Semi-Supervised Learning of GMM

Video-driven state-aware facial animation

Example-Based Automatic Music-Driven Conventional Dance Motion Synthesis

Automatic 3D Motion Synthesis with Time-Striding Hidden Markov Model.

Composable Semi parametric Modelling for Long range Motion Generation

Editing and Synthesis of 3D Facial Animation by Motion Propagation and Isomap Analysis

HuMoR: 3D Human Motion Model for Robust Pose Estimation

Human Motion Generation with Multifactor Models

Animatable Virtual Humans: Learning pose-dependent human representations in UV space for interactive performance synthesis

Make-It-Animatable: An Efficient Framework for Authoring Animation-Ready 3D Characters

A Survey on Machine Learning in the Synthesis of Human Motions

Towards 3D Dance Motion Synthesis and Control