Abstract:Talking face generation is the process of synthesizing a lip-synchronized video when given a reference portrait and an audio clip. However, generating a fine-grained talking video is nontrivial due to several challenges: 1) capturing vivid facial expressions, such as muscle movements; 2) ensuring smooth transitions between consecutive frames; and 3) preserving the details of the reference portrait. Existing efforts have only focused on modeling rigid lip movements, resulting in low-fidelity videos with jerky facial muscle deformations. To address these challenges, we propose a novel Fine-gRained mOtioN moDel (FROND), consisting of three components. In the first component, we adopt a two-stream encoder to capture local facial movement keypoints and embed their overall motion context as the global code. In the second component, we design a motion estimation module to predict audio-driven movements. This enables the learning of local key point motion in the continuous trajectory space to achieve smooth temporal facial movements. Additionally, the local and global motions are fused to estimate a continuous dense motion field, resulting in spatially smooth movements. In the third component, we devise a novel implicit image decoder based on an implicit neural network. This decoder recovers high-frequency information from the input image, resulting in a high-fidelity talking face. In summary, the FROND refines the motion trajectories of facial keypoints into a continuous dense motion field, which is followed by a decoder that fully exploits the inherent smoothness of the motion. We conduct quantitative and qualitative model evaluations on benchmark datasets. The experimental results show that our proposed FROND significantly outperforms several state-of-the-art baselines.

Animating Portrait Line Drawings from a Single Face Photo and a Speech Signal

Audio-driven Talking Face Video Generation with Natural Head Pose

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation

Generating Animatable 3D Cartoon Faces from Single Portraits

Realistic Dynamic Facial Textures from a Single Image Using GANs

MyPortrait: Morphable Prior-Guided Personalized Portrait Generation

MakeItTalk: Speaker-Aware Talking-Head Animation

Realistic Speech-Driven Facial Animation with GANs

Line Drawings for Face Portraits from Photos Using Global and Local Structure Based GANs

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

Toward Fine-Grained Talking Face Generation

Apdrawinggan: Generating Artistic Portrait Drawings From Face Photos With Hierarchical Gans

OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow Guidance

Quality Metric Guided Portrait Line Drawing Generation from Unpaired Training Data

VectorTalker: SVG Talking Face Generation with Progressive Vectorisation

Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

Generative Adversarial Talking Head: Bringing Portraits to Life with a Weakly Supervised Neural Network

Controllable Talking Face Generation by Implicit Facial Keypoints Editing

AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

Language-Guided Face Animation by Recurrent StyleGAN-based Generator