Abstract:Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.

Lip Animation System for Visual Speech Synthesis in Uyghur Language

Expressive lip-synchronization facial animation system

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

Attention-Based Lip Audio-Visual Synthesis for Talking Face Generation in the Wild

3D Facial Animation from Chinese Text.

Lip-movement synthesis from speech based on CDHMM-SVR

Lip syncing method for realistic expressive 3D face model

Audio-to-Deep-Lip: Speaking Lip Synthesis Based on 3D Landmarks

Lip Movement Synthesis in Audio-Visual Speech Recognition System

Individual facial image synthesis system for a virtual human

A Synthesis Method of Three-dimensional Facial Animation with Multiple Elements Blending Based on MPEG-4

A Novel Speech-Driven Lip-Sync Model with CNN and LSTM

Speech driven photo-realistic face animation with mouth and jaw dynamics

Lip Movement and Expression Database Construction for Chinese Sign Language Synthesis

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video

Expressive facial animation synthesis by learning speech coarticulation and expression spaces

Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation

VividWav2Lip: High-Fidelity Facial Animation Generation Based on Speech-Driven Lip Synchronization

Expressive Face Animation Synthesis Based on Dynamic Mapping Method

HyperLips: Hyper Control Lips with High Resolution Decoder for Talking Face Generation