High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

Weizhi Zhong,Junfan Lin,Peixin Chen,Liang Lin,Guanbin Li

2024-08-10

Abstract:Audio-driven talking face video generation has attracted increasing attention due to its huge industrial potential. Some previous methods focus on learning a direct mapping from audio to visual content. Despite progress, they often struggle with the ambiguity of the mapping process, leading to flawed results. An alternative strategy involves facial structural representations (e.g., facial landmarks) as intermediaries. This multi-stage approach better preserves the appearance details but suffers from error accumulation due to the independent optimization of different stages. Moreover, most previous methods rely on generative adversarial networks, prone to training instability and mode collapse. To address these challenges, our study proposes a novel landmark-based diffusion model for talking face generation, which leverages facial landmarks as intermediate representations while enabling end-to-end optimization. Specifically, we first establish the less ambiguous mapping from audio to landmark motion of lip and jaw. Then, we introduce an innovative conditioning module called TalkFormer to align the synthesized motion with the motion represented by landmarks via differentiable cross-attention, which enables end-to-end optimization for improved lip synchronization. Besides, TalkFormer employs implicit feature warping to align the reference image features with the target motion for preserving more appearance details. Extensive experiments demonstrate that our approach can synthesize high-fidelity and lip-synced talking face videos, preserving more subject appearance details from the reference image.

Computer Vision and Pattern Recognition,Artificial Intelligence,Multimedia

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to generate high - fidelity and lip - synchronized talking - face videos. Specifically, the author aims to overcome the following challenges in existing methods for audio - driven talking - face video generation:

1. **Ambiguity of direct mapping**: Some previous methods focus on learning the direct mapping from audio to visual content, but this mapping process often has ambiguity, resulting in unsatisfactory results.
2. **Error accumulation in multi - stage methods**: Another strategy is to use facial structure representations (such as facial key points) as intermediaries. Although this method can better preserve appearance details, error accumulation is likely to occur due to independent optimization at different stages.
3. **Training instability and mode collapse of generative adversarial networks (GANs)**: Most previous methods rely on generative adversarial networks, which makes the training process unstable and prone to mode - collapse problems.

To solve these problems, the author proposes a landmark - based diffusion model, which uses facial key points as an intermediate representation and simultaneously achieves end - to - end optimization. In this way, the model can reduce the error accumulation caused by inaccurate key - point estimation and thus generate higher - quality and lip - synchronized talking - face videos.

### Main contributions of the paper

1. **Proposing a novel method**: By using facial key points as an intermediate representation to simplify the ambiguity of audio - visual mapping and minimizing error accumulation through end - to - end optimization, high - fidelity and lip - synchronized talking - face videos are generated.
2. **Introducing a new conditional module, TalkFormer**: This module aligns the synthesized actions with the actions represented by key points through a differentiable cross - attention mechanism, achieving joint optimization at different stages. In addition, TalkFormer also aligns the reference image features based on semantic relevance, enhancing the preservation of appearance details.
3. **Extensive experimental verification**: Through a large number of experiments, the effectiveness of this method in generating high - quality and lip - synchronized talking - face videos has been verified, and it can be generalized to any unseen subject without additional fine - tuning.

### Formula summary

- **Loss function**:
\[
L_{\text{total}} = L_{\text{ldm}}+\lambda L_1
\]
where \(L_{\text{ldm}}\) is the denoising objective function, defined as follows:
\[
L_{\text{ldm}}=\mathbb{E}_{z, \epsilon\sim\mathcal{N}(0, 1), t}\left[\left\|\epsilon-\epsilon_\theta(z_t, t)\right\|_2^2\right]
\]
\(L_1\) is the loss function of the landmark completion module, defined as follows:
\[
L_1=\sum_{i = 0}^{L - 1}\left(\left\|\hat{C}_{\text{lip}}^{\tau + i}-C_{\text{lip}}^{\tau + i}\right\|_1+\left\|\hat{C}_{\text{jaw}}^{\tau + i}-C_{\text{jaw}}^{\tau + i}\right\|_1\right)
\]

These formulas ensure that the model can effectively optimize landmark prediction and the denoising process during training, thereby improving the lip - synchronization effect.