Abstract:Audio-driven one-shot talking face generation methods are usually trained on video resources of various persons. However, their created videos often suffer unnatural mouth shapes and asynchronous lips because those methods struggle to learn a consistent speech style from different speakers. We observe that it would be much easier to learn a consistent speech style from a specific speaker, which leads to authentic mouth movements. Hence, we propose a novel one-shot talking face generation framework by exploring consistent correlations between audio and visual motions from a specific speaker and then transferring audio-driven motion fields to a reference image. Specifically, we develop an Audio-Visual Correlation Transformer (AVCT) that aims to infer talking motions represented by keypoint based dense motion fields from an input audio. In particular, considering audio may come from different identities in deployment, we incorporate phonemes to represent audio signals. In this manner, our AVCT can inherently generalize to audio spoken by other identities. Moreover, as face keypoints are used to represent speakers, AVCT is agnostic against appearances of the training speaker, and thus allows us to manipulate face images of different identities readily. Considering different face shapes lead to different motions, a motion field transfer module is exploited to reduce the audio-driven dense motion field gap between the training identity and the one-shot reference. Once we obtained the dense motion field of the reference image, we employ an image renderer to generate its talking face videos from an audio clip. Thanks to our learned consistent speaking style, our method generates authentic mouth shapes and vivid movements. Extensive experiments demonstrate that our synthesized videos outperform the state-of-the-art in terms of visual quality and lip-sync.

You Said That?: Synthesising Talking Faces from Audio

Audio-driven Talking Face Video Generation with Natural Head Pose

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Talking face generation driven by time-frequency domain features of speech audio

FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning

That's What I Said: Fully-Controllable Talking Face Generation

Talking Face Video Generation with Editable Expression

Talking Faces: Audio-to-Video Face Generation

Meta Talk: Learning To Data-Efficiently Generate Audio-Driven Lip-Synchronized Talking Face With High Definition

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

Multimodal Inputs Driven Talking Face Generation With Spatial–Temporal Dependency

Attention-Based Lip Audio-Visual Synthesis for Talking Face Generation in the Wild

VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild

Talking Face Generation With Audio-Deduced Emotional Landmarks

JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation

Wav2Lip-HR: Synthesising Clear High-Resolution Talking Head in the Wild

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Persons

Controllable Talking Face Generation by Implicit Facial Keypoints Editing

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

Dynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressions

One-Shot Talking Face Generation from Single-Speaker Audio-Visual Correlation Learning