Audio-Driven 3D Facial Animation from In-the-Wild Videos

Liying Lu,Tianke Zhang,Yunfei Liu,Xuangeng Chu,Yu Li

2023-06-20

Abstract:Given an arbitrary audio clip, audio-driven 3D facial animation aims to generate lifelike lip motions and facial expressions for a 3D head. Existing methods typically rely on training their models using limited public 3D datasets that contain a restricted number of audio-3D scan pairs. Consequently, their generalization capability remains limited. In this paper, we propose a novel method that leverages in-the-wild 2D talking-head videos to train our 3D facial animation model. The abundance of easily accessible 2D talking-head videos equips our model with a robust generalization capability. By combining these videos with existing 3D face reconstruction methods, our model excels in generating consistent and high-fidelity lip synchronization. Additionally, our model proficiently captures the speaking styles of different individuals, allowing it to generate 3D talking-heads with distinct personal styles. Extensive qualitative and quantitative experimental results demonstrate the superiority of our method.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of audio-driven 3D facial animation generation, with specific objectives including:

1. **Improving the model's generalization ability**: Existing methods typically rely on limited publicly available 3D datasets for training, which contain a small number of audio-3D scan pairs, resulting in limited generalization ability of the models. This paper proposes a method to train 3D facial animation models using a large amount of easily accessible in-the-wild 2D talking head videos to enhance the model's generalization ability.

2. **Generating high-fidelity lip-sync effects**: By combining existing 3D face reconstruction methods, the model can produce consistent and high-precision lip-sync effects across different individuals.

3. **Capturing different speaking styles**: The model can capture the speaking styles of different individuals and generate 3D talking head videos with specific identity characteristics.

4. **Emotion control functionality**: The model also has emotion control capabilities, allowing it to generate corresponding expressions based on specified emotional styles (such as anger, happiness, or sadness).

In summary, the main contribution of the paper is the proposal of a new method that utilizes in-the-wild 2D video data to train 3D facial animation models, thereby addressing the data scarcity issue in existing methods and significantly improving the model's performance in generating realistic and accurately lip-synced 3D facial animations.