Abstract:We introduce a novel method for synthesizing dance motions that follow the emotions and contents of a piece of music. Our method employs a learning-based approach to model the music to motion mapping relationship embodied in example dance motions along with those motions' accompanying background music. A key step in our method is to train a music to motion matching quality rating function through learning the music to motion mapping relationship exhibited in synchronized music and dance motion data, which were captured from professional human dance performance. To generate an optimal sequence of dance motion segments to match with a piece of music, we introduce a constraint-based dynamic programming procedure. This procedure considers both music to motion matching quality and visual smoothness of a resultant dance motion sequence. We also introduce a two-way evaluation strategy, coupled with a GPU-based implementation, through which we can execute the dynamic programming process in parallel, resulting in significant speedup. To evaluate the effectiveness of our method, we quantitatively compare the dance motions synthesized by our method with motion synthesis results by several peer methods using the motions captured from professional human dancers' performance as the gold standard. We also conducted several medium-scale user studies to explore how perceptually our dance motion synthesis method can outperform existing methods in synthesizing dance motions to match with a piece of music. These user studies produced very positive results on our music-driven dance motion synthesis experiments for several Asian dance genres, confirming the advantages of our method.

Real-Time Music Synthesis by Patch-Based Method

audeosynth: music-driven video montage

SongDriver2: Real-time Emotion-based Music Arrangement with Soft Transition.

Example-Based Automatic Music-Driven Conventional Dance Motion Synthesis

Real-time texture synthesis by patch-based sampling

A Texture Synthesis Method Based on Adaptive Patches

A New Method of Music Melody Extraction and Its Application

GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework

Real-Time Texture Synthesis with Patch Jump Maps

Exploring Tokenization Methods for Multitrack Sheet Music Generation

A Method for Detecting Short Similar Segments in Music Borrowing

InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models

Simple and Controllable Music Generation

Seed-Music: A Unified Framework for High Quality and Controlled Music Generation

A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis

Hierarchical Timbre-Painting and Articulation Generation

SynthNet: Learning to Synthesize Music End-to-End

Transformer-XL Based Music Generation with Multiple Sequences of Time-valued Notes

Analyzing and reducing the synthetic-to-real transfer gap in Music Information Retrieval: the task of automatic drum transcription

Long-Term Rhythmic Video Soundtracker