Abstract:Audio-driven talking face video generation has attracted increasing attention due to its huge industrial potential. Some previous methods focus on learning a direct mapping from audio to visual content. Despite progress, they often struggle with the ambiguity of the mapping process, leading to flawed results. An alternative strategy involves facial structural representations (e.g., facial landmarks) as intermediaries. This multi-stage approach better preserves the appearance details but suffers from error accumulation due to the independent optimization of different stages. Moreover, most previous methods rely on generative adversarial networks, prone to training instability and mode collapse. To address these challenges, our study proposes a novel landmark-based diffusion model for talking face generation, which leverages facial landmarks as intermediate representations while enabling end-to-end optimization. Specifically, we first establish the less ambiguous mapping from audio to landmark motion of lip and jaw. Then, we introduce an innovative conditioning module called TalkFormer to align the synthesized motion with the motion represented by landmarks via differentiable cross-attention, which enables end-to-end optimization for improved lip synchronization. Besides, TalkFormer employs implicit feature warping to align the reference image features with the target motion for preserving more appearance details. Extensive experiments demonstrate that our approach can synthesize high-fidelity and lip-synced talking face videos, preserving more subject appearance details from the reference image.

Audio to Deep Visual: Speaking Mouth Generation Based on 3D Sparse Landmarks

Audio-driven Talking Face Video Generation with Natural Head Pose

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

Audio-Driven 3D Facial Animation from In-the-Wild Videos

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

Learning Landmarks Motion from Speech for Speaker-Agnostic 3D Talking Heads Generation

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Experts

TalkingFlow: Talking Facial Landmark Generation with Multi-Scale Normalizing Flow Network

Acoustic VR in the Mouth: A Real-Time Speech-Driven Visual Tongue System.

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

A Novel Speech-Driven Lip-Sync Model with CNN and LSTM

Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation

PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis

Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation

KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding

Generating Holistic 3D Human Motion from Speech

Audio-Driven Emotional 3D Talking-Head Generation

Spatially and Temporally Optimized Audio‐Driven Talking Face Generation

SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces

Generating Talking Face Landmarks from Speech