Abstract:In this work, we propose a method to transform a speaker’s speech information into a target character’s talking video; the method could make the mouth shape synchronization, expression, and body posture more realistic in the synthesized speaker video. This is a challenging task because changes of mouth shape and posture are coupled with audio semantic information. The model training is difficult to converge, and the model effect is unstable in complex scenes. Existing speech-driven speaker methods cannot solve this problem well. The method proposed in this paper first generates the sequence of key points of the speaker’s face and body postures from the audio signal in real time and then visualizes these key points as a series of two-dimensional skeleton images. Subsequently, we generate the final real speaker video through the video generation network. We take a random sampling of audio clips, encode audio contents and temporal correlations using a more effective network structure, and optimize and iterate network outputs using differential loss and attitude perception loss, so as to obtain a smoother pose key-point sequence and better performance. In addition, by inserting a specified action frame into the synthesized human pose sequence window, action poses of the synthesized speaker are enriched, making the synthesis effect more realistic and natural. Then, the final speaker video is generated by the obtained gesture key points through the video generation network. In order to generate realistic and high-resolution pose detail videos, we insert a local attention mechanism into the key point network of the generated pose sequence and give higher attention to the local details of the characters through spatial weight masks. In order to verify the effectiveness of the proposed method, we used the objective evaluation index NME and user subjective evaluation methods, respectively. Experiment results showed that our method could vividly use audio contentsto generate corresponding speaker videos, and its lip-matching accuracy and expression postures are better than those of previous work. Compared with existing methods in the NME index and user subjective evaluation, our method showed better results.

Learn to Gesture - Let Your Body Speak.

GestureLens: Visual Analysis of Gestures in Presentation Videos

LLM Knows Body Language, Too: Translating Speech Voices into Human Gestures

Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents

Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis

Editable Co-Speech Gesture Synthesis Enhanced with Individual Representative Gestures

Learning Speech-driven 3D Conversational Gestures from Video

ExpressGesture: Expressive gesture generation from speech through database matching

Emphasizing Semantic Consistency of Salient Posture for Speech-Driven Gesture Generation

Prediction-Based Gesture Detection in Lecture Videos by Combining Visual, Speech and Electronic Slides

Audio-driven Neural Gesture Reenactment with Video Motion Graphs

Realistic Speech-Driven Talking Video Generation with Personalized Pose

GestureMaster: Graph-based Speech-driven Gesture Generation

Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Audio-Driven Co-Speech Gesture Video Generation

Deep Gesture Generation for Social Robots Using Type-Specific Libraries

Slideshow: Gesture-aware PPT Presentation

SIGGesture: Generalized Co-Speech Gesture Synthesis via Semantic Injection with Large-Scale Pre-Training Diffusion Models

Simulating a Smartboard by Real-Time Gesture Detection in Lecture Videos