Abstract:In this work, we propose a method to transform a speaker’s speech information into a target character’s talking video; the method could make the mouth shape synchronization, expression, and body posture more realistic in the synthesized speaker video. This is a challenging task because changes of mouth shape and posture are coupled with audio semantic information. The model training is difficult to converge, and the model effect is unstable in complex scenes. Existing speech-driven speaker methods cannot solve this problem well. The method proposed in this paper first generates the sequence of key points of the speaker’s face and body postures from the audio signal in real time and then visualizes these key points as a series of two-dimensional skeleton images. Subsequently, we generate the final real speaker video through the video generation network. We take a random sampling of audio clips, encode audio contents and temporal correlations using a more effective network structure, and optimize and iterate network outputs using differential loss and attitude perception loss, so as to obtain a smoother pose key-point sequence and better performance. In addition, by inserting a specified action frame into the synthesized human pose sequence window, action poses of the synthesized speaker are enriched, making the synthesis effect more realistic and natural. Then, the final speaker video is generated by the obtained gesture key points through the video generation network. In order to generate realistic and high-resolution pose detail videos, we insert a local attention mechanism into the key point network of the generated pose sequence and give higher attention to the local details of the characters through spatial weight masks. In order to verify the effectiveness of the proposed method, we used the objective evaluation index NME and user subjective evaluation methods, respectively. Experiment results showed that our method could vividly use audio contentsto generate corresponding speaker videos, and its lip-matching accuracy and expression postures are better than those of previous work. Compared with existing methods in the NME index and user subjective evaluation, our method showed better results.

Audio-Semantic Enhanced Pose-Driven Talking Head Generation

Audio-driven Talking Face Video Generation with Natural Head Pose

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

Towards Realistic Conversational Head Generation: A Comprehensive Framework for Lifelike Video Synthesis

Realistic Speech-Driven Talking Video Generation with Personalized Pose

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Audio-Driven Emotional 3D Talking-Head Generation

MakeItTalk: Speaker-Aware Talking-Head Animation

PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation

SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation.

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

High-Fidelity and Freely Controllable Talking Head Video Generation

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation

Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Experts

Audio-Driven Talking Head Video Generation with Diffusion Model

Listen, Disentangle, and Control: Controllable Speech-Driven Talking Head Generation

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Persons

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation

Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation