Abstract:Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.

Research on face video generation algorithm based on speech content

AudioVSR: Enhancing Video Speech Recognition with Audio Data

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

APB2FaceV2: Real-Time Audio-Guided Multi-Face Reenactment

Audio-driven Talking Face Video Generation with Natural Head Pose

Meta Talk: Learning To Data-Efficiently Generate Audio-Driven Lip-Synchronized Talking Face With High Definition

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert

Wavsyncswap: End-To-End Portrait-Customized Audio-Driven Talking Face Generation

LaDTalk: Latent Denoising for Synthesizing Talking Head Videos with High Frequency Details

VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild

VividWav2Lip: High-Fidelity Facial Animation Generation Based on Speech-Driven Lip Synchronization

SATFace: Subject Agnostic Talking Face Generation with Natural Head Movement

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video

Talking face generation driven by time-frequency domain features of speech audio

Talking Faces: Audio-to-Video Face Generation

A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model

RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network

SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Space

Speech Fusion to Face: Bridging the Gap Between Human's Vocal Characteristics and Facial Imaging

G4G:A Generic Framework for High Fidelity Talking Face Generation with Fine-grained Intra-modal Alignment