Abstract:Real-world talking faces often accompany with natural head movement. However, most existing talking face video generation methods only consider facial animation with fixed head pose. In this paper, we address this problem by proposing a deep neural network model that takes an audio signal A of a source person and a very short video V of a target person as input, and outputs a synthesized high-quality talking face video with personalized head pose (making use of the visual information in V), expression and lip synchronization (by considering both A and V). The most challenging issue in our work is that natural poses often cause in-plane and out-of-plane head rotations, which makes synthesized talking face video far from realistic. To address this challenge, we reconstruct 3D face animation and re-render it into synthesized frames. To fine tune these frames into realistic ones with smooth background transition, we propose a novel memory-augmented GAN module. By first training a general mapping based on a publicly available dataset and fine-tuning the mapping using the input short video of target person, we develop an effective strategy that only requires a small number of frames (about 300 frames) to learn personalized talking behavior including head pose. Extensive experiments and two user studies show that our method can generate high-quality (i.e., personalized head movements, expressions and good lip synchronization) talking face videos, which are naturally looking with more distinguishing head movement effects than the state-of-the-art methods.

3D Facial Expression Synthesis Based on Nonlinear Co-learning

Personalized 3-D facial expression synthesis based on landmark constraint.

GPU-Accelerated Parallel Realistic 3D Facial Expression Synthesis

Coupled Dictionary Learning for the Detail-Enhanced Synthesis of 3-D Facial Expressions

Joint sparse learning for 3-D facial expression generation.

Facial Landmarks and Expression Label Guided Photorealistic Facial Expression Synthesis

Sparse coding for flexible, robust 3D facial-expression synthesis.

Facial Expression Synthesis Using Manifold Learning and Belief Propagation

Video-driven state-aware facial animation

Audio-driven Talking Face Video Generation with Natural Head Pose

A Unified Deep Model for Joint Facial Expression Recognition, Face Synthesis, and Face Alignment.

3D Facial Modeling for Animation: A Nonlinear Approach

3D Animation Automatic Generation System Design Based on Deep Learning

Expressive facial animation synthesis by learning speech coarticulation and expression spaces

End-to-end Learning for 3D Facial Animation from Raw Waveforms of Speech

Facial Expression Synthesis and Recognition Using a Kernel-Based Factorization Model

3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head

Expressive 3D Facial Animation Generation Based on Local-to-Global Latent Diffusion

Data-driven facial expression synthesis via Laplacian deformation

CSTalk: Correlation Supervised Speech-driven 3D Emotional Facial Animation Generation