Abstract:3D Facial animations, crucial to augmented and mixed reality digital media, have evolved from mere aesthetic elements to potent storytelling media. Despite considerable progress in facial animation of neutral emotions, existing methods still struggle to capture the authenticity of emotions. This paper introduces a novel approach to capture fine facial expressions and generate facial animations using audio synchronization. Our method consists of two key components: First, the Local-to-global Latent Diffusion Model (LG-LDM) tailored for authentic facial expressions, which can integrate audio, time step, facial expressions, and other conditions towards possible encoding of emotionally rich yet latent features in response to possibly noisy raw audio signals. The core of LG-LDM is our carefully designed Facial Denoiser Model (FDM) for aligning the local-to-global animation feature with audio. Second, we redesign an Emotion-centric Vector Quantized-Variational AutoEncoder framework (EVQ-VAE) to finely decode the subtle differences under different emotions and reconstruct the final 3D facial geometry. Our work significantly contributes to the key challenges of emotionally realistic 3D facial animation for audio synchronization and enhances the immersive experience and emotional depth in augmented and mixed reality applications. We provide a reproducibility kit including our code, dataset, and detailed instructions for running the experiments. This kit is available at https://github.com/wangxuanx/Face-Diffusion-Model.

Realistic 3d Facial Animations With Subtle Texture Changes

Realistic 3D Facial Animations with Partial Expression Ratio Image

Video-Based Facial Animation with Detailed Appearance Texture

GPU-Accelerated Parallel Realistic 3D Facial Expression Synthesis

Quick Customization of Particular Human Face and Muscle-Driven Expression Animation

Personalized Multi-View Face Animation with Lifelike Textures

Facial Expression Animation Based on Physical Model

Performance-Driven Animation of Hand-Drawn Cartoon Faces

3D Facial Animation from Chinese Text.

A Piece-Wise Learning Approach To 3d Facial Animation

Expressive 3D Facial Animation Generation Based on Local-to-Global Latent Diffusion

An Efficient Approach for Multi-View Face Animation Based on Quasi 3d Model

GENERATION OF REALISTIC 3-D SPECIFIC HUMAN FACE

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

Research on 3D Facial Expression Simulation Technology

EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

Facial Expression Synthesis Based on Emotion Dimensions for Affective Talking Avatar

Real-time Synthesis of Chinese Visual Speech and Facial Expressions Using MPEG-4 FAP Features in a Three-Dimensional Avatar

Modeling short-term dynamics and variability for realistic interactive facial animation

AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation