Abstract:We present DanceAnyWay, a generative learning method to synthesize beat-guided dances of 3D human characters synchronized with music. Our method learns to disentangle the dance movements at the beat frames from the dance movements at all the remaining frames by operating at two hierarchical levels. At the coarser "beat" level, it encodes the rhythm, pitch, and melody information of the input music via dedicated feature representations only at the beat frames. It leverages them to synthesize the beat poses of the target dances using a sequence-to-sequence learning framework. At the finer "repletion" level, our method encodes similar rhythm, pitch, and melody information from all the frames of the input music via dedicated feature representations. It generates the full dance sequences by combining the synthesized beat and repletion poses and enforcing plausibility through an adversarial learning framework. Our training paradigm also enforces fine-grained diversity in the synthesized dances through a randomized temporal contrastive loss, which ensures different segments of the dance sequences have different movements and avoids motion freezing or collapsing to repetitive movements. We evaluate the performance of our approach through extensive experiments on the benchmark AIST++ dataset and observe improvements of about 7%-12% in motion quality metrics and 1.5%-4% in motion diversity metrics over the current baselines, respectively. We also conducted a user study to evaluate the visual quality of our synthesized dances. We note that, on average, the samples generated by our method were about 9-48% more preferred by the participants and had a 4-27% better five-point Likert-scale score over the best available current baseline in terms of motion quality and synchronization. Our source code and project page are available at <a class="link-external link-https" href="https://github.com/aneeshbhattacharya/DanceAnyWay" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper aims to address the problem of generating 3D dance motion sequences synchronized with music, particularly focusing on ensuring that the generated motions follow the music's rhythm and maintain diversity in long sequences to avoid repetitive patterns. To achieve this goal, the paper proposes a method called DanceAnyWay, which is implemented through two levels of learning: 1. **Beat Pose Synthesis (BPS)**: At a coarser level, this method focuses on extracting rhythm, pitch, and melody information from the audio and encoding it into so-called "beat frames." The dance motions on these keyframes are referred to as "beat poses," which form the basic structure of the entire dance. The goal of the BPS stage is to generate dance motions on these keyframes to capture the overall dance characteristics. 2. **Repletion Pose Synthesis (RPS)**: At a finer level, the task of the RPS stage is to generate dance motions on all remaining frames, known as "repletion poses." This part ensures smooth transitions between dance motions and guarantees the reasonableness of the generated motions through an adversarial learning framework. Additionally, the paper proposes a Randomized Temporal Contrastive Loss to enforce motion diversity during the generation process, especially in long sequences, to avoid motion freezing or repetition. In summary, the main contributions of DanceAnyWay include: - Proposing a hierarchical learning method based on sequence-to-sequence learning and generative adversarial networks for synthesizing beat-aligned 3D dance sequences synchronized with audio. - Introducing a Randomized Temporal Contrastive Loss to generate motions with fine-grained diversity, particularly in long sequences. - Utilizing a spatiotemporal graph representation of 3D human poses to effectively learn local (joint-level) and macro (body-level) motions in different dances. - Providing an end-to-end audio-to-dance generation pipeline that demonstrates state-of-the-art performance in multiple quantitative and qualitative evaluations.

DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning

Example-Based Automatic Music-Driven Conventional Dance Motion Synthesis

Robust Dancer: Long-term 3D Dance Synthesis Using Unpaired Data

DeepDance: Music-to-Dance Motion Choreography With Adversarial Learning

May the Dance be with You: Dance Generation Framework for Non-Humanoids

Music2Dance: DanceNet for Music-Driven Dance Generation

Dance Any Beat: Blending Beats with Visuals in Dance Video Generation

Feel The Music: Automatically Generating A Dance For An Input Song

Beat-It: Beat-Synchronized Multi-Condition 3D Dance Generation

DanceNet3D: Music Based Dance Generation with Parametric Motion Transformer

DanceIt: Music-Inspired Dancing Video Synthesis

Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning

DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer

Towards 3D Dance Motion Synthesis and Control

Music2Dance: Music-driven Dance Generation Using WaveNet.

Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio

PC-Dance: Posture-controllable Music-driven Dance Synthesis

Music-oriented Dance Video Synthesis with Pose Perceptual Loss