MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting

Yue Zhang,Minhao Liu,Zhaokang Chen,Bin Wu,Yubin Zeng,Chao Zhan,Yingjie He,Junxin Huang,Wenjiang Zhou

2024-10-16

Abstract:Achieving high-resolution, identity consistency, and accurate lip-speech synchronization in face visual dubbing presents significant challenges, particularly for real-time applications like live video streaming. We propose MuseTalk, which generates lip-sync targets in a latent space encoded by a Variational Autoencoder, enabling high-fidelity talking face video generation with efficient inference. Specifically, we project the occluded lower half of the face image and itself as an reference into a low-dimensional latent space and use a multi-scale U-Net to fuse audio and visual features at various levels. We further propose a novel sampling strategy during training, which selects reference images with head poses closely matching the target, allowing the model to focus on precise lip movement by filtering out redundant information. Additionally, we analyze the mechanism of lip-sync loss and reveal its relationship with input information volume. Extensive experiments show that MuseTalk consistently outperforms recent state-of-the-art methods in visual fidelity and achieves comparable lip-sync accuracy. As MuseTalk supports the online generation of face at 256x256 at more than 30 FPS with negligible starting latency, it paves the way for real-time applications.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of generating high-resolution, identity-consistent, and lip-synced facial visual dubbing videos in real-time applications. Specifically, the paper focuses on the following three main challenges:

1. **High Resolution**: The generated facial video needs to maintain high resolution to ensure the authenticity and detail of the visual effects.
2. **Identity Consistency**: The generated facial video needs to maintain identity consistency with the original face, avoiding distortion of facial features due to errors in the generation process.
3. **Lip Syncing**: The generated facial video needs to be highly synchronized with the input audio, ensuring that lip movements precisely match the speech content.

These challenges are particularly significant in real-time applications, such as live video streaming. Existing methods have some shortcomings in addressing these issues, such as low generation quality, high computational resource consumption, and slow inference speed. Therefore, this paper proposes a new method called MuseTalk, which aims to solve the above problems by performing lip inpainting in the latent space, achieving efficient and high-quality real-time facial visual dubbing.