Conditional GAN for Enhancing Diffusion Models in Efficient and Authentic Global Gesture Generation from Audios

Yongkang Cheng,Mingjiang Liang,Shaoli Huang,Gaoge Han,Jifeng Ning,Wei Liu

2024-11-01

Abstract:Audio-driven simultaneous gesture generation is vital for human-computer communication, AI games, and film production. While previous research has shown promise, there are still limitations. Methods based on VAEs are accompanied by issues of local jitter and global instability, whereas methods based on diffusion models are hampered by low generation efficiency. This is because the denoising process of DDPM in the latter relies on the assumption that the noise added at each step is sampled from a unimodal distribution, and the noise values are small. DDIM borrows the idea from the Euler method for solving differential equations, disrupts the Markov chain process, and increases the noise step size to reduce the number of denoising steps, thereby accelerating generation. However, simply increasing the step size during the step-by-step denoising process causes the results to gradually deviate from the original data distribution, leading to a significant drop in the quality of the generated actions and the emergence of unnatural artifacts. In this paper, we break the assumptions of DDPM and achieves breakthrough progress in denoising speed and fidelity. Specifically, we introduce a conditional GAN to capture audio control signals and implicitly match the multimodal denoising distribution between the diffusion and denoising steps within the same sampling step, aiming to sample larger noise values and apply fewer denoising steps for high-speed generation.

Sound,Artificial Intelligence,Computer Vision and Pattern Recognition,Graphics,Audio and Speech Processing

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the limitations in efficiency and quality of audio - driven simultaneous gesture generation. Specifically:

1. **Methods based on variational auto - encoder (VAE)**: These methods will have problems of local jitter and global instability during the generation process.
2. **Methods based on diffusion models**: Although these methods perform well in terms of generation quality, their generation efficiency is low. For example, DiffGesture takes 6 seconds to generate an 88 - frame gesture sequence segment, which is unacceptable in real - time applications.

To overcome these problems, the author proposes a new method to improve the denoising speed and fidelity of the diffusion model by introducing conditional generative adversarial network (Conditional GAN). Specific improvements include:

- **Introducing conditional GAN**: Capture audio control signals and implicitly match the multi - modal denoising distribution between diffusion and denoising steps within the same sampling step, so as to achieve high - speed generation in fewer denoising steps.
- **Explicit motion geometry loss**: In order to generate high - fidelity global gestures and avoid unnatural artifacts, an explicit motion geometry loss is introduced.

Through these improvements, the author claims that their method is significantly superior to existing diffusion model methods in both generation speed and generation quality, and performs well in real - time tasks.