Abstract:People may perform diverse gestures affected by various mental and physical factors when speaking the same sentences. This inherent one-to-many relationship makes co-speech gesture generation from audio particularly challenging. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, easily resulting in plain/boring motions during inference. So we propose to explicitly model the one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code is expected to be responsible for the motion component that is more correlated to the audio while the motion-specific code is expected to capture diverse motion information that is more independent of the audio. However, splitting the latent code into two parts poses extra training difficulties. Several crucial training losses/strategies, including relaxed motion loss, bicycle constraint, and diversity loss, are designed to better train the VAE. Experiments on both 3D and 2D motion datasets verify that our method generates more realistic and diverse motions than previous state-of-the-art methods, quantitatively and qualitatively. Besides, our formulation is compatible with discrete cosine transformation (DCT) modeling and other popular backbones (i.e., RNN, Transformer). As for motion losses and quantitative motion evaluation, we find structured losses/metrics (e.g. STFT) that consider temporal and/or spatial context complement the most commonly used point-wise losses (e.g. PCK), resulting in better motion dynamics and more nuanced motion details. Finally, we demonstrate that our method can be readily used to generate motion sequences with user-specified motion clips on the timeline.

Evaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM Network

Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

Analyzing Input and Output Representations for Speech-Driven Gesture Generation

Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents

Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning

Deep Gesture Generation for Social Robots Using Type-Specific Libraries

EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation

Audio-driven Neural Gesture Reenactment with Video Motion Graphs

Audio-Driven Co-Speech Gesture Video Generation

DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework

Learning to gesticulate by observation using a deep generative approach

Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis

Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation

Audio2Gestures: Generating Diverse Gestures From Audio

Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks

Audio2Face: Generating Speech/Face Animation from Single Audio with Attention-Based Bidirectional LSTM Networks

Learning Speech-driven 3D Conversational Gestures from Video

Gesticulator: A framework for semantically-aware speech-driven gesture generation

Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation