Abstract:Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.

Movement epenthesis with prosody control for Chinese sign language video synthesis

Text-driven Visual Prosody Generation for Embodied Conversational Agents

Human-Robot Sign Language Motion Retargeting from Videos

Research on the improved gesture tracking algorithm in sign language synthesis

Head Movement Synthesis Based on Semantic and Prosodic Features for a Chinese Expressive Avatar

A Method to Synthesize Chinese Sign Language Based on Virtual Human Technologies

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation

Pose-Guided Fine-Grained Sign Language Video Generation

Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observation

Modeling the Speed and Timing of American Sign Language to Generate Realistic Animations

Expressive Face Animation Synthesis Based on Dynamic Mapping Method

LinguaLinker: Audio-Driven Portraits Animation with Implicit Facial Control Enhancement

SpeechAct: Towards Generating Whole-body Motion from Speech

Expressive facial animation synthesis by learning speech coarticulation and expression spaces

Audio-Driven Co-Speech Gesture Video Generation

Modeling Incompletion Phenomenon in Mandarin Dialog Prosody.

Stylized Synthesis of Facial Speech Motions

HandTalker II: a Chinese sign language recognition and synthesis system

Talking-head Generation with Rhythmic Head Motion

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video