Abstract:Real-world talking faces often accompany with natural head movement. However, most existing talking face video generation methods only consider facial animation with fixed head pose. In this paper, we address this problem by proposing a deep neural network model that takes an audio signal A of a source person and a very short video V of a target person as input, and outputs a synthesized high-quality talking face video with personalized head pose (making use of the visual information in V), expression and lip synchronization (by considering both A and V). The most challenging issue in our work is that natural poses often cause in-plane and out-of-plane head rotations, which makes synthesized talking face video far from realistic. To address this challenge, we reconstruct 3D face animation and re-render it into synthesized frames. To fine tune these frames into realistic ones with smooth background transition, we propose a novel memory-augmented GAN module. By first training a general mapping based on a publicly available dataset and fine-tuning the mapping using the input short video of target person, we develop an effective strategy that only requires a small number of frames (about 300 frames) to learn personalized talking behavior including head pose. Extensive experiments and two user studies show that our method can generate high-quality (i.e., personalized head movements, expressions and good lip synchronization) talking face videos, which are naturally looking with more distinguishing head movement effects than the state-of-the-art methods.

Emotional Chinese talking head system

Text-driven Visual Prosody Generation for Embodied Conversational Agents

Audio-Driven Emotional 3D Talking-Head Generation

An Online Speech Driven Talking Head System

A Computational Model of Emotion for 3D Talking Heads

Emotional Talking Agent: System and Evaluation

3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head

Visual Speech Emotion Conversion Using Deep Learning for 3D Talking Head

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Head

Audio-driven Talking Face Video Generation with Natural Head Pose

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation

Facial Animation System for Embedded Application

Animating a Chinese interactive virtual character

Listen, Disentangle, and Control: Controllable Speech-Driven Talking Head Generation

Mood avatar: automatic text-driven head motion synthesis

Facial Expression Synthesis Based on Emotion Dimensions for Affective Talking Avatar

Real-time Speech-Driven Animation of Expressive Talking Faces.

From Talking Head To Singing Head: A Significant Enhancement For More Natural Human Computer Interaction

Emotional Talking Head Generation based on Memory-Sharing and Attention-Augmented Networks

Voicing Your Emotion: Integrating Emotion and Identity in Cross-Modal 3D Facial Animations