Abstract:Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.

Learning based visual speech synthesis system

Visual Speech Synthesis Based on Learning Model

Bi-level Codebook Based Speech-driven Visual-speech Synthesis System

Lip Movement Synthesis in Audio-Visual Speech Recognition System

GA-Based Speaking Mouth Correlative Speech Feature Abstraction

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video

Realistic Visual Speech Synthesis Based on Hybrid Concatenation Method

Attention-Based Lip Audio-Visual Synthesis for Talking Face Generation in the Wild

Lipsynthesis Incorporating Audio-Visual Synchronisation

A Survey on Speech-synch Visual Speech Synthesizing Techniques

Extracting Lip Parameters in Speech Synthesis System Driven by Visual-Speech

VISUAL SPEECH SYNTHESIS BASED ON ARTICULATORY TRAJECTORY

Dynamic mapping method based speech driven face animation system

Visualtts: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

LPIPS-AttnWav2Lip: Generic Audio-Driven Lip Synchronization for Talking Head Generation in the Wild

Dynamic Audio-Visual Mapping using Fused Hidden Markov Model Inversion Method

Joint Audio-Video Driven Facial Animation

Visual Speech Animation

Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert

Lip Movements Generation at a Glance