Abstract:In this work, we propose a method to transform a speaker’s speech information into a target character’s talking video; the method could make the mouth shape synchronization, expression, and body posture more realistic in the synthesized speaker video. This is a challenging task because changes of mouth shape and posture are coupled with audio semantic information. The model training is difficult to converge, and the model effect is unstable in complex scenes. Existing speech-driven speaker methods cannot solve this problem well. The method proposed in this paper first generates the sequence of key points of the speaker’s face and body postures from the audio signal in real time and then visualizes these key points as a series of two-dimensional skeleton images. Subsequently, we generate the final real speaker video through the video generation network. We take a random sampling of audio clips, encode audio contents and temporal correlations using a more effective network structure, and optimize and iterate network outputs using differential loss and attitude perception loss, so as to obtain a smoother pose key-point sequence and better performance. In addition, by inserting a specified action frame into the synthesized human pose sequence window, action poses of the synthesized speaker are enriched, making the synthesis effect more realistic and natural. Then, the final speaker video is generated by the obtained gesture key points through the video generation network. In order to generate realistic and high-resolution pose detail videos, we insert a local attention mechanism into the key point network of the generated pose sequence and give higher attention to the local details of the characters through spatial weight masks. In order to verify the effectiveness of the proposed method, we used the objective evaluation index NME and user subjective evaluation methods, respectively. Experiment results showed that our method could vividly use audio contentsto generate corresponding speaker videos, and its lip-matching accuracy and expression postures are better than those of previous work. Compared with existing methods in the NME index and user subjective evaluation, our method showed better results.

Speech driven photo-realistic face animation with mouth and jaw dynamics

Audio-driven Talking Face Video Generation with Natural Head Pose

Speech driven photo realistic facial animation based on an articulatory DBN model and AAM features

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation

Individual 3D Face Synthesis Based on Orthogonal Photos and Speech-Driven Facial Animation

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

Video-audio Driven Real-Time Facial Animation.

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

MakeItTalk: Speaker-Aware Talking-Head Animation

Real-time Speech-Driven Animation of Expressive Talking Faces.

Realistic talking face animation with speech-induced head motion

Realistic Speech-Driven Talking Video Generation with Personalized Pose

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

Neural Face Models for Example-Based Visual Speech Synthesis

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Persons

Speech4Mesh: Speech-Assisted Monocular 3D Facial Reconstruction for Speech-Driven 3D Facial Animation

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

Multimodal Learning for Temporally Coherent Talking Face Generation with Articulator Synergy

SATFace: Subject Agnostic Talking Face Generation with Natural Head Movement