Abstract:Generating vivid and diverse 3D co-speech gestures is crucial for various applications in animating virtual avatars. While most existing methods can generate gestures from audio directly, they usually overlook that emotion is one of the key factors of authentic co-speech gesture generation. In this work, we propose EmotionGesture, a novel framework for synthesizing vivid and diverse emotional co-speech 3D gestures from audio. Considering emotion is often entangled with the rhythmic beat in speech audio, we first develop an Emotion-Beat Mining module (EBM) to extract the emotion and audio beat features as well as model their correlation via a transcript-based visual-rhythm alignment. Then, we propose an initial pose based Spatial-Temporal Prompter (STP) to generate future gestures from the given initial poses. STP effectively models the spatial-temporal correlations between the initial poses and the future gestures, thus producing the spatial-temporal coherent pose prompt. Once we obtain pose prompts, emotion, and audio beat features, we will generate 3D co-speech gestures through a transformer architecture. However, considering the poses of existing datasets often contain jittering effects, this would lead to generating unstable gestures. To address this issue, we propose an effective objective function, dubbed Motion-Smooth Loss. Specifically, we model motion offset to compensate for jittering ground-truth by forcing gestures to be smooth. Last, we present an emotion-conditioned VAE to sample emotion features, enabling us to generate diverse emotional results. Extensive experiments demonstrate that our framework outperforms the state-of-the-art, achieving vivid and diverse emotional co-speech 3D gestures. Our code and dataset will be released at the project page: <a class="link-external link-https" href="https://xingqunqi-lab.github.io/Emotion-Gesture-Web/" rel="external noopener nofollow">this https URL</a>

Adversarial gesture generation with realistic gesture phasing

Speech gesture generation from the trimodal context of text, audio, and speaker identity

Investigating the use of recurrent motion modelling for speech gesture generation

ExpressGesture: Expressive gesture generation from speech through database matching

Analyzing Input and Output Representations for Speech-Driven Gesture Generation

Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents

The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settings

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning

Gesticulator: A framework for semantically-aware speech-driven gesture generation

Learning Speech-driven 3D Conversational Gestures from Video

A Framework for Integrating Gesture Generation Models into Interactive Conversational Agents

DeepNAG: Deep Non-Adversarial Gesture Generation

Co-Speech Gesture Synthesis using Discrete Gesture Token Learning

Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance

Audio-Driven Co-Speech Gesture Video Generation

EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation

GestureMaster: Graph-based Speech-driven Gesture Generation

Learning to generate pointing gestures in situated embodied conversational agents