Abstract:3D Facial animations, crucial to augmented and mixed reality digital media, have evolved from mere aesthetic elements to potent storytelling media. Despite considerable progress in facial animation of neutral emotions, existing methods still struggle to capture the authenticity of emotions. This paper introduces a novel approach to capture fine facial expressions and generate facial animations using audio synchronization. Our method consists of two key components: First, the Local-to-global Latent Diffusion Model (LG-LDM) tailored for authentic facial expressions, which can integrate audio, time step, facial expressions, and other conditions towards possible encoding of emotionally rich yet latent features in response to possibly noisy raw audio signals. The core of LG-LDM is our carefully designed Facial Denoiser Model (FDM) for aligning the local-to-global animation feature with audio. Second, we redesign an Emotion-centric Vector Quantized-Variational AutoEncoder framework (EVQ-VAE) to finely decode the subtle differences under different emotions and reconstruct the final 3D facial geometry. Our work significantly contributes to the key challenges of emotionally realistic 3D facial animation for audio synchronization and enhances the immersive experience and emotional depth in augmented and mixed reality applications. We provide a reproducibility kit including our code, dataset, and detailed instructions for running the experiments. This kit is available at https://github.com/wangxuanx/Face-Diffusion-Model.

Realistic 3D Facial Animations with Partial Expression Ratio Image

Realistic 3d Facial Animations With Subtle Texture Changes

Video-Based Facial Animation with Detailed Appearance Texture

GPU-Accelerated Parallel Realistic 3D Facial Expression Synthesis

Facial Expression Animation Based on Physical Model

Quick Customization of Particular Human Face and Muscle-Driven Expression Animation

Performance-Driven Animation of Hand-Drawn Cartoon Faces

Personalized Multi-View Face Animation with Lifelike Textures

Research on 3D Facial Expression Simulation Technology

Video Tracked Facial Expression Animation

MPEG-4 Based Facial Expression Image Morphing

GENERATION OF REALISTIC 3-D SPECIFIC HUMAN FACE

Facial Expression Synthesis Based on Emotion Dimensions for Affective Talking Avatar

Facial Expression Synthesis Using PAD Emotional Parameters for a Chinese Expressive Avatar

Image Ratio Features for Facial Expression Recognition Application

An Efficient Approach for Multi-View Face Animation Based on Quasi 3d Model

Automatic Face Modeling and Synthesis Based on Image Pairs

A Piece-Wise Learning Approach To 3d Facial Animation

Expressive 3D Facial Animation Generation Based on Local-to-Global Latent Diffusion

3D Facial Animation from Chinese Text.