Abstract:Real-world talking faces often accompany with natural head movement. However, most existing talking face video generation methods only consider facial animation with fixed head pose. In this paper, we address this problem by proposing a deep neural network model that takes an audio signal A of a source person and a very short video V of a target person as input, and outputs a synthesized high-quality talking face video with personalized head pose (making use of the visual information in V), expression and lip synchronization (by considering both A and V). The most challenging issue in our work is that natural poses often cause in-plane and out-of-plane head rotations, which makes synthesized talking face video far from realistic. To address this challenge, we reconstruct 3D face animation and re-render it into synthesized frames. To fine tune these frames into realistic ones with smooth background transition, we propose a novel memory-augmented GAN module. By first training a general mapping based on a publicly available dataset and fine-tuning the mapping using the input short video of target person, we develop an effective strategy that only requires a small number of frames (about 300 frames) to learn personalized talking behavior including head pose. Extensive experiments and two user studies show that our method can generate high-quality (i.e., personalized head movements, expressions and good lip synchronization) talking face videos, which are naturally looking with more distinguishing head movement effects than the state-of-the-art methods.

Speech Driven Face Animation based on Dynamic Concatenation Model

Dynamic mapping method based speech driven face animation system

Realistic Visual Speech Synthesis Based on Hybrid Concatenation Method

Text-driven Visual Prosody Generation for Embodied Conversational Agents

Expressive Face Animation Synthesis Based on Dynamic Mapping Method

Audio-driven Talking Face Video Generation with Natural Head Pose

Real-time Speech-Driven Animation of Expressive Talking Faces.

Stylized Synthesis of Facial Speech Motions

3D Realistic Talking Face Co-Driven by Text and Speech

Speech driven photo-realistic face animation with mouth and jaw dynamics

Speech Driven Facial Animation Using Chinese Mandarin Pronunciation Rules

3D Visible Speech Animation Driven by Prosody Text

Speech-Driven 3D Face Animation with Composite and Regional Facial Movements

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

Speech-driven Facial Animation with Spectral Gathering and Temporal Attention.

Brand-new Speech Animation Technology Based on First Order Motion Model and MelGAN-VC

Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation

Joint Audio-Video Driven Facial Animation

Face Animation Based on Large Audiovisual Database

Face and Voice Synchronization Animation System Based on Chinese Text Drive

A Speech-Driven Lip Synchronization Method