Abstract:We introduce a novel method for synthesizing dance motions that follow the emotions and contents of a piece of music. Our method employs a learning-based approach to model the music to motion mapping relationship embodied in example dance motions along with those motions' accompanying background music. A key step in our method is to train a music to motion matching quality rating function through learning the music to motion mapping relationship exhibited in synchronized music and dance motion data, which were captured from professional human dance performance. To generate an optimal sequence of dance motion segments to match with a piece of music, we introduce a constraint-based dynamic programming procedure. This procedure considers both music to motion matching quality and visual smoothness of a resultant dance motion sequence. We also introduce a two-way evaluation strategy, coupled with a GPU-based implementation, through which we can execute the dynamic programming process in parallel, resulting in significant speedup. To evaluate the effectiveness of our method, we quantitatively compare the dance motions synthesized by our method with motion synthesis results by several peer methods using the motions captured from professional human dancers' performance as the gold standard. We also conducted several medium-scale user studies to explore how perceptually our dance motion synthesis method can outperform existing methods in synthesizing dance motions to match with a piece of music. These user studies produced very positive results on our music-driven dance motion synthesis experiments for several Asian dance genres, confirming the advantages of our method.

A Motion-Driven System for Performing Art

A Motion-based User Interface for the Control of Virtual Humans Performing Sports.

Responsive Action Generation By Physically-Based Motion Retrieval And Adaptation

Motion-Based Perceptual User Interface

Real-Time Motion Capture Technology on a Live Theatrical Performance with Computer Generated Scenery

A Penetrable Interactive 3D Display Based on Motion Recognition （invited Paper）

Example-Based Automatic Music-Driven Conventional Dance Motion Synthesis

Building Virtual Entertainment Environment with Tiled Display Wall and Motion Tracking.

A Real-Time Interactive System of Surface Reconstruction and Dynamic Projection Mapping with RGB-depth Sensor and Projector.

Performance-Driven Motion Retrieval and its Usability Evaluation

Choreographing the Digital Canvas: A Machine Learning Approach to Artistic Performance

Research and realization of stage augmented reality system

Automatic Authoring of Physical and Perceptual/Affective Motion Effects for Virtual Reality

Interactive Animation of Virtual Characters: Application to Virtual Kung-Fu Fighting

Physical World to Virtual Reality – Motion Capture Technology in Dance Creation

Research on the Application of Wireless Wearable Sensing Devices in Interactive Music

A Real-Time Articulated Human Motion Tracking Using Tri-Axis Inertial/magnetic Sensors Package.

Real-Time Human Motion Transfer System for Holographic Displays

Everything2Motion: Synchronizing Diverse Inputs Via a Unified Framework for Human Motion Synthesis

A Human-Computer Duet System for Music Performance

HapMotion: motion-to-tactile framework with wearable haptic devices for immersive VR performance experience