Abstract:Audio-driven one-shot talking face generation methods are usually trained on video resources of various persons. However, their created videos often suffer unnatural mouth shapes and asynchronous lips because those methods struggle to learn a consistent speech style from different speakers. We observe that it would be much easier to learn a consistent speech style from a specific speaker, which leads to authentic mouth movements. Hence, we propose a novel one-shot talking face generation framework by exploring consistent correlations between audio and visual motions from a specific speaker and then transferring audio-driven motion fields to a reference image. Specifically, we develop an Audio-Visual Correlation Transformer (AVCT) that aims to infer talking motions represented by keypoint based dense motion fields from an input audio. In particular, considering audio may come from different identities in deployment, we incorporate phonemes to represent audio signals. In this manner, our AVCT can inherently generalize to audio spoken by other identities. Moreover, as face keypoints are used to represent speakers, AVCT is agnostic against appearances of the training speaker, and thus allows us to manipulate face images of different identities readily. Considering different face shapes lead to different motions, a motion field transfer module is exploited to reduce the audio-driven dense motion field gap between the training identity and the one-shot reference. Once we obtained the dense motion field of the reference image, we employ an image renderer to generate its talking face videos from an audio clip. Thanks to our learned consistent speaking style, our method generates authentic mouth shapes and vivid movements. Extensive experiments demonstrate that our synthesized videos outperform the state-of-the-art in terms of visual quality and lip-sync.

Adaptive Super Resolution For One-Shot Talking-Head Generation

Audio-driven Talking Face Video Generation with Natural Head Pose

High-Fidelity and Freely Controllable Talking Head Video Generation

Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

Robust One Shot Audio to Video Generation

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

Generating Smooth and Facial-Details-Enhanced Talking Head Video: A Perspective of Pre and Post Processes

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

One-Shot Talking Face Generation from Single-Speaker Audio-Visual Correlation Learning

Audio-Semantic Enhanced Pose-Driven Talking Head Generation

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Persons

Wav2Lip-HR: Synthesising Clear High-Resolution Talking Head in the Wild

OPT: One-shot Pose-Controllable Talking Head Generation

StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation

Speech Driven Talking Head Generation Via Attentional Landmarks Based Representation

StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles

Realistic talking face synthesis with geometry-aware feature transformation