EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions

Zhiyuan Chen,Jiajiong Cao,Zhiquan Chen,Yuming Li,Chenguang Ma

2024-07-12

Abstract:The area of portrait image animation, propelled by audio input, has witnessed notable progress in the generation of lifelike and dynamic portraits. Conventional methods are limited to utilizing either audios or facial key points to drive images into videos, while they can yield satisfactory results, certain issues exist. For instance, methods driven solely by audios can be unstable at times due to the relatively weaker audio signal, while methods driven exclusively by facial key points, although more stable in driving, can result in unnatural outcomes due to the excessive control of key point information. In addressing the previously mentioned challenges, in this paper, we introduce a novel approach which we named EchoMimic. EchoMimic is concurrently trained using both audios and facial landmarks. Through the implementation of a novel training strategy, EchoMimic is capable of generating portrait videos not only by audios and facial landmarks individually, but also by a combination of both audios and selected facial landmarks. EchoMimic has been comprehensively compared with alternative algorithms across various public datasets and our collected dataset, showcasing superior performance in both quantitative and qualitative evaluations. Additional visualization and access to the source code can be located on the EchoMimic project page.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address several key issues in portrait animation generation, particularly those encountered in audio-driven portrait animation (i.e., "talking head" animation). Specifically, traditional portrait animation methods typically rely solely on audio signals or facial keypoints to drive the image-to-video conversion, each with its own limitations:

1. **Methods relying solely on audio**: While they can generate dynamic portrait videos, the relatively weak audio signal may lead to unstable results in some cases.
2. **Methods relying solely on facial keypoints**: Although the driving process is relatively stable, over-reliance on keypoint information may result in unnatural generation outcomes.

To address these issues, the paper proposes a new method—EchoMimic. EchoMimic utilizes both audio signals and facial keypoints for training, and through a novel training strategy, the model can generate portrait videos using either audio or facial keypoints alone, or by combining both to drive video generation. This method has been comprehensively compared on multiple public datasets, demonstrating its superior performance in both quantitative and qualitative evaluations.

Overall, the goal of the paper is to improve the stability and naturalness of portrait animation while maintaining the authenticity and dynamism of the portrait.