Abstract:This paper addresses the problem of generating lifelike holistic co-speech motions for 3D avatars, focusing on two key aspects: variability and coordination. Variability allows the avatar to exhibit a wide range of motions even with similar speech content, while coordination ensures a harmonious alignment among facial expressions, hand gestures, and body poses. We aim to achieve both with ProbTalk, a unified probabilistic framework designed to jointly model facial, hand, and body movements in speech. ProbTalk builds on the variational autoencoder (VAE) architecture and incorporates three core designs. First, we introduce product quantization (PQ) to the VAE, which enriches the representation of complex holistic motion. Second, we devise a novel non-autoregressive model that embeds 2D positional encoding into the product-quantized representation, thereby preserving essential structure information of the PQ codes. Last, we employ a secondary stage to refine the preliminary prediction, further sharpening the high-frequency details. Coupling these three designs enables ProbTalk to generate natural and diverse holistic co-speech motions, outperforming several state-of-the-art methods in qualitative and quantitative evaluations, particularly in terms of realism. Our code and model will be released for research purposes at
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to generate realistic holistic co - speech motions, with a particular focus on two key aspects: diversity and coordination. Diversity allows virtual characters to exhibit a wide range of motions under similar speech content, while coordination ensures harmony among facial expressions, gestures, and body postures. Specifically, the authors propose a unified probabilistic framework named ProbTalk, aiming to jointly model facial, hand, and body motions from speech to achieve the dual goals of diversity and coordination.
### Background of the Paper and Problem Definition
The paper points out that communication is not just about what is heard, but a comprehensive sensory experience that integrates non - verbal signals such as body postures, gestures, and facial expressions. Therefore, automatically generating realistic full - body motions synchronized with speech is crucial for providing a more immersive and interactive user experience. However, earlier methods often led to repetitive and less human - like motions because the same speech content would produce the same motions. To solve this problem, the paper proposes ProbTalk, a novel framework based on the variational auto - encoder (VAE) architecture, which generates natural and diverse holistic co - speech motions through three core designs:
1. **Product Quantization (PQ)**: Divide the latent space of holistic motions into multiple sub - spaces for independent quantization, thereby enriching the representation of complex holistic motions.
2. **Non - autoregressive model**: Combine MaskGIT and 2D position encoding to effectively preserve the 2D structural information of PQ codes and significantly reduce the inference steps.
3. **Second - stage refinement**: Further sharpen the high - frequency details of the preliminary prediction, especially in facial motions.
### Method Overview
The paper's method mainly includes the following parts:
- **PQ - VAE**: By dividing the latent space into multiple sub - spaces and performing independent quantization, PQ - VAE can represent complex holistic motions with lower quantization error.
- **Non - autoregressive model**: Using MaskGIT and 2D position encoding, this model can efficiently predict diverse body motions.
- **Motion detail optimization**: Further optimize the preliminarily generated motions through a sequence - to - sequence model, capture finer details, and improve the synchronization accuracy with audio.
### Experimental Results
The paper demonstrates the superior performance of ProbTalk in generating realistic and diverse holistic co - speech motions through quantitative and qualitative evaluation methods. Specific indicators include:
- **Fréchet Gesture Distance (FGD)**: Measures the difference between the generated motion and the real - motion distribution.
- **Variance**: Evaluates the diversity of generated samples under the same conditions.
- **Frames per second (FPS)**: Measures the inference efficiency.
The experimental results show that ProbTalk outperforms all existing state - of - the - art methods in all FGD indicators, especially in generating realistic holistic body motions and diverse facial expressions. In addition, ProbTalk also has an advantage in inference efficiency.
### Conclusion
The ProbTalk framework proposed in the paper successfully solves the problems of diversity and coordination in generating realistic holistic co - speech motions, providing a new solution for the natural interaction of virtual characters.