Generating Holistic 3D Human Motion from Speech

Hongwei Yi,Hualin Liang,Yifei Liu,Qiong Cao,Yandong Wen,Timo Bolkart,Dacheng Tao,Michael J. Black

2023-06-18

Abstract:This work addresses the problem of generating 3D holistic body motions from human speech. Given a speech recording, we synthesize sequences of 3D body poses, hand gestures, and facial expressions that are realistic and diverse. To achieve this, we first build a high-quality dataset of 3D holistic body meshes with synchronous speech. We then define a novel speech-to-motion generation framework in which the face, body, and hands are modeled separately. The separated modeling stems from the fact that face articulation strongly correlates with human speech, while body poses and hand gestures are less correlated. Specifically, we employ an autoencoder for face motions, and a compositional vector-quantized variational autoencoder (VQ-VAE) for the body and hand motions. The compositional VQ-VAE is key to generating diverse results. Additionally, we propose a cross-conditional autoregressive model that generates body poses and hand gestures, leading to coherent and realistic motions. Extensive experiments and user studies demonstrate that our proposed approach achieves state-of-the-art performance both qualitatively and quantitatively. Our novel dataset and code will be released for research purposes at <a class="link-external link-https" href="https://talkshow.is.tue.mpg.de" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The paper aims to address the problem of generating 3D full-body motions from human speech, including body postures, gestures, and facial expressions. Specifically, the goal of the research is to synthesize a series of realistic and diverse 3D body postures, gestures, and facial expressions from a segment of speech recording.

To achieve this goal, the authors first constructed a high-quality dataset that includes 3D full-body meshes and synchronized speech signals. Then, they proposed a novel speech-to-motion generation framework, where the face, body, and hands are modeled separately. Considering that facial movements are highly correlated with speech signals, while body postures and gestures are less correlated, the authors employed different techniques to handle these different body parts. For facial movements, a simple autoencoder-based structure was used to encode rich phoneme information; for body and hand movements, a composite Vector Quantized Variational Autoencoder (VQ-VAE) was designed to capture diverse motion patterns.

Additionally, the paper proposed a cross-conditional autoregressive model to generate body postures and gestures, ensuring that the generated motions are both coherent and realistic. Experimental results show that the proposed method achieves state-of-the-art performance in both qualitative and quantitative evaluations, and the dataset and code have been made publicly available for research purposes.