Abstract:Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.

Robust Lip Localization on Multi-View Faces in Video

Robust Human Tracking Via Key Face Information.

An Approach to Robust and Fast Locating of Lip Motion

Lip Reading Based on 3D Face Modeling and Spatial Transformation Learning

2-Entity Random Sample Consensus for Robust Visual Localization: Framework, Methods, and Verifications

Recurrent Volume-based 3D Feature Fusion for Real-time Multi-view Object Pose Estimation

Robust Speaking Face Identification For Video Analysis

Robust Video Identification Approach Based on Local Non-Negative Matrix Factorization

Lip Contour Extraction Based on Support Vector Machine Add Support

Robust Lip Segmentation Based on Complexion Mixture Model.

Robust 3D face landmark localization based on local coordinate coding.

Robust 3D Face Recognition by Local Shape Difference Boosting

Multi-View Exclusive Unsupervised Dimension Reduction for Video-Based Facial Expression Recognition

Joint Structured Sparsity Regularized Multiview Dimension Reduction for Video-Based Facial Expression Recognition.

A probabilistic model for robust face alignment in videos

Multi-Grained Spatio-temporal Modeling for Lip-reading

LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video

Robust video fingerprinting based on visual attention regions

Robust Face Alignment Based on Local Texture Classifiers

Robust Head Pose Estimation Using LGBP