LinguaLinker: Audio-Driven Portraits Animation with Implicit Facial Control Enhancement

Rui Zhang,Yixiao Fang,Zhengnan Lu,Pei Cheng,Zebiao Huang,Bin Fu

2024-07-26

Abstract:This study delves into the intricacies of synchronizing facial dynamics with multilingual audio inputs, focusing on the creation of visually compelling, time-synchronized animations through diffusion-based techniques. Diverging from traditional parametric models for facial animation, our approach, termed LinguaLinker, adopts a holistic diffusion-based framework that integrates audio-driven visual synthesis to enhance the synergy between auditory stimuli and visual responses. We process audio features separately and derive the corresponding control gates, which implicitly govern the movements in the mouth, eyes, and head, irrespective of the portrait's origin. The advanced audio-driven visual synthesis mechanism provides nuanced control but keeps the compatibility of output video and input audio, allowing for a more tailored and effective portrayal of distinct personas across different languages. The significant improvements in the fidelity of animated portraits, the accuracy of lip-syncing, and the appropriate motion variations achieved by our method render it a versatile tool for animating any portrait in any language.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main goal of this paper is to address the issue of synchronizing multilingual audio input with facial dynamics, focusing on creating visually engaging and temporally synchronized animations through diffusion model techniques. Specifically, the paper proposes a method called LinguaLinker, which is a diffusion model-based holistic framework capable of generating high-fidelity, lip-synced animated portrait videos that support multilingual audio input. LinguaLinker implicitly controls the movements of the lips, eyes, and head by processing audio features and deriving corresponding control gates, regardless of the portrait's origin. Additionally, this method provides fine-grained control and maintains compatibility between the output video and the input audio, allowing characters in different languages to be more personalized and effectively presented.

In the experimental section, the paper demonstrates LinguaLinker's performance on multiple datasets and compares it with other methods. The results show that LinguaLinker achieves significant improvements in the realism of generated results, lip-sync accuracy, and appropriate motion variations, making it a versatile tool for animating any portrait in any language. However, the paper also points out some limitations of this method, including the lengthy inference process and slight differences in lip-sync performance across different languages. Future research directions will focus on addressing these issues to achieve more refined generation results.