Abstract:Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.

Lip Viseme Analysis of Chinese Shaanxi Xi’an Dialect Visual Speech for Talking Head in Speech Assistant System

Text-driven Visual Prosody Generation for Embodied Conversational Agents

Learning Audio-Driven Viseme Dynamics for 3D Face Animation

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

Lip Assistant: Visualize Speech For Hearing Impaired People In Multimedia Services

VisemeNet: Audio-Driven Animator-Centric Speech Animation

Real-time Synthesis of Chinese Visua using MPEG-4 FAP Features in a

Real-time Synthesis of Chinese Visual Speech and Facial Expressions Using MPEG-4 FAP Features in a Three-Dimensional Avatar

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video

SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces

Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert

Speaker-independent Lips and Tongue Visualization of Vowels

Study of Chinese Viseme

Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert

VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild

Head Movement Synthesis Based on Semantic and Prosodic Features for a Chinese Expressive Avatar

An Online Speech Driven Talking Head System

Learn2Talk: 3D Talking Face Learns from 2D Talking Face

Audio-Driven 3D Facial Animation from In-the-Wild Videos

PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features

MMHead: Towards Fine-grained Multi-modal 3D Facial Animation