Abstract:Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.

Audio–video syncing with lip movements using generative deep neural networks

VividWav2Lip: High-Fidelity Facial Animation Generation Based on Speech-Driven Lip Synchronization

A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild

Audio-driven Talking Face Generation with Stabilized Synchronization Loss

Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization

Lip Movements Generation at a Glance

LaDTalk: Latent Denoising for Synthesizing Talking Head Videos with High Frequency Details

VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices

SyncTalklip: Highly Synchronized Lip-Readable Speaker Generation with Multi-Task Learning

LawDNet: Enhanced Audio-Driven Lip Synthesis via Local Affine Warping Deformation

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video

A Neural Lip-Sync Framework for Synthesizing Photorealistic Virtual News Anchors

Real-time Lip Synchronization Based on Hidden Markov Models

Learning Speaker-specific Lip-to-Speech Generation

StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator

ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer

MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting

SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis

Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis

A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation