Abstract:Audio-driven talking face video generation has attracted increasing attention due to its huge industrial potential. Some previous methods focus on learning a direct mapping from audio to visual content. Despite progress, they often struggle with the ambiguity of the mapping process, leading to flawed results. An alternative strategy involves facial structural representations (e.g., facial landmarks) as intermediaries. This multi-stage approach better preserves the appearance details but suffers from error accumulation due to the independent optimization of different stages. Moreover, most previous methods rely on generative adversarial networks, prone to training instability and mode collapse. To address these challenges, our study proposes a novel landmark-based diffusion model for talking face generation, which leverages facial landmarks as intermediate representations while enabling end-to-end optimization. Specifically, we first establish the less ambiguous mapping from audio to landmark motion of lip and jaw. Then, we introduce an innovative conditioning module called TalkFormer to align the synthesized motion with the motion represented by landmarks via differentiable cross-attention, which enables end-to-end optimization for improved lip synchronization. Besides, TalkFormer employs implicit feature warping to align the reference image features with the target motion for preserving more appearance details. Extensive experiments demonstrate that our approach can synthesize high-fidelity and lip-synced talking face videos, preserving more subject appearance details from the reference image.

Personalized 3-D facial expression synthesis based on landmark constraint.

3D Facial Expression Synthesis Based on Nonlinear Co-learning

GPU-Accelerated Parallel Realistic 3D Facial Expression Synthesis

Facial Landmarks and Expression Label Guided Photorealistic Facial Expression Synthesis

Video-driven state-aware facial animation

Audio-driven Talking Face Video Generation with Natural Head Pose

Video Tracked Facial Expression Animation

LBF Based 3D Regression for Facial Animation

Coupled Dictionary Learning for the Detail-Enhanced Synthesis of 3-D Facial Expressions

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation

Expressive 3D face synthesis by multi-space modeling

Data-driven facial expression synthesis via Laplacian deformation

Robust 3D face landmark localization based on local coordinate coding.

CSTalk: Correlation Supervised Speech-driven 3D Emotional Facial Animation Generation

Sparse coding for flexible, robust 3D facial-expression synthesis.

Expressive facial animation synthesis by learning speech coarticulation and expression spaces

Joint sparse learning for 3-D facial expression generation.

Enhancing Facial Consistency in Conditional Video Generation via Facial Landmark Transformation

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model