Abstract:Speech-driven facial animation is useful for a variety of applications such as telepresence, chatbots, etc. The necessary attributes of having a realistic face animation are 1) audio-visual synchronization (2) identity preservation of the target individual (3) plausible mouth movements (4) presence of natural eye blinks. The existing methods mostly address the audio-visual lip synchronization, and few recent works have addressed the synthesis of natural eye blinks for overall video realism. In this paper, we propose a method for identity-preserving realistic facial animation from speech. We first generate person-independent facial landmarks from audio using DeepSpeech features for invariance to different voices, accents, etc. To add realism, we impose eye blinks on facial landmarks using unsupervised learning and retargets the person-independent landmarks to person-specific landmarks to preserve the identity-related facial structure which helps in the generation of plausible mouth shapes of the target identity. Finally, we use LSGAN to generate the facial texture from person-specific facial landmarks, using an attention mechanism that helps to preserve identity-related texture. An extensive comparison of our proposed method with the current state-of-the-art methods demonstrates a significant improvement in terms of lip synchronization accuracy, image reconstruction quality, sharpness, and identity-preservation. A user study also reveals improved realism of our animation results over the state-of-the-art methods. To the best of our knowledge, this is the first work in speech-driven 2D facial animation that simultaneously addresses all the above-mentioned attributes of a realistic speech-driven face animation.

Realistic talking face animation with speech-induced head motion

Audio-driven Talking Face Video Generation with Natural Head Pose

Identity-Preserving Realistic Talking Face Generation

Realistic Speech-Driven Facial Animation with GANs

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

Talking-head Generation with Rhythmic Head Motion

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation

FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models

MakeItTalk: Speaker-Aware Talking-Head Animation

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

SATFace: Subject Agnostic Talking Face Generation with Natural Head Movement

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

Speech-Driven Facial Animations Improve Speech-in-Noise Comprehension of Humans

A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Listen, Disentangle, and Control: Controllable Speech-Driven Talking Head Generation

High-Fidelity and Freely Controllable Talking Head Video Generation

Predicting Personalized Head Movement From Short Video and Speech Signal

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

3D Realistic Talking Face Co-Driven by Text and Speech