Abstract:Real-world talking faces often accompany with natural head movement. However, most existing talking face video generation methods only consider facial animation with fixed head pose. In this paper, we address this problem by proposing a deep neural network model that takes an audio signal A of a source person and a very short video V of a target person as input, and outputs a synthesized high-quality talking face video with personalized head pose (making use of the visual information in V), expression and lip synchronization (by considering both A and V). The most challenging issue in our work is that natural poses often cause in-plane and out-of-plane head rotations, which makes synthesized talking face video far from realistic. To address this challenge, we reconstruct 3D face animation and re-render it into synthesized frames. To fine tune these frames into realistic ones with smooth background transition, we propose a novel memory-augmented GAN module. By first training a general mapping based on a publicly available dataset and fine-tuning the mapping using the input short video of target person, we develop an effective strategy that only requires a small number of frames (about 300 frames) to learn personalized talking behavior including head pose. Extensive experiments and two user studies show that our method can generate high-quality (i.e., personalized head movements, expressions and good lip synchronization) talking face videos, which are naturally looking with more distinguishing head movement effects than the state-of-the-art methods.

SPECIFIC FACE SYNTHESIS BASED ON THE MODIFICATION OF THE GENERAL FACE MODEL

The Synthesis Of Realistic Human Face

GENERATION OF REALISTIC 3-D SPECIFIC HUMAN FACE

Facial Expression Synthesis Based on Facial Component Model.

An experimental facial synthesis system using graph cut and gradient domain fusion

Specific Face Model and Expression Animation Model Based on Physical Model

Fast face modeling based on features and flexible mesh model

Facial Expression Animation Based on Physical Model

Individual facial image synthesis system for a virtual human

Quick Customization of Particular Human Face and Muscle-Driven Expression Animation

Realistic 3-D human face synthesis based on a deformable face model

Synthesis of Combined Facial Expressions Using Anatomy-based Model

Audio-driven Talking Face Video Generation with Natural Head Pose

Extended facial expression synthesis using statistical appearance model

Expression Synthesis Method Based on Single Facial Image

Active appearance model-based expression synthesis method

A Facial Expression Model For Human-Like Agent

A Novel Frontal Facial Synthesis Algorithm Based on Individual Residual Face.

Realistic Tree-Dimensional Facial Expression Synthesis

Three DimensionaI FaciaI Expression Synthesis Based on a SingIe Photo

Deformable model-based generation of realistic 3-D specific human face