ExpAvatar: High-Fidelity Avatar Generation of Unseen Expressions with 3D Face Priors

Yuan Gan,Ruijie Quan,Yawei Luo

DOI: https://doi.org/10.1145/3700770

2024-01-01

Abstract:The reconstruction of dynamic head avatars has gained increasing significance, giving rise to various downstream applications such as visual dubbing and digital human creation. Despite recent advancements, generating novel, unseen expressions for a given identity remains challenging in concurrently achieving 1) accurate expression and consistent appearance and 2) high-quality and realistic faces. This paper introduces ExpAvatar, a novel approach crafted to address these challenges. ExpAvatar elaborately leverages the appearance consistency capabilities inherent in 3DMMs-based models along with the robust generalization ability of DDPMs-based models to alleviate appearance drift issues and enhance the generation of unseen expressions. Specifically, ExpAvatar introduces a Face Priors-conditioned Diffusion model (FPDiff) to inject 3D face priors into generation models through fine-tuning. Furthermore, a Face Priors-conditioned Catalyst (FPCatalyst) is employed to enhance the inference efficiency and generation quality. Moreover, we propose a unique confidence-based regularizer function to mitigate the effect of imperfect face-tracking estimates, thereby improving the quality of dynamic neural head avatars. Experimental results demonstrate that ExpAvatar surpasses current state-of-the-art solutions in generating unseen expressions, marking an advancement in the realm of dynamic head avatar synthesis. Code: https://github.com/yuangan/ExpAvatar.