Abstract:Audio-driven 3D facial animation synthesis has been an active field of research with attention from both academia and industry. While there are promising results in this area, recent approaches largely focus on lip-sync and identity control, neglecting the role of emotions and emotion control in the generative process. That is mainly due to the lack of emotionally rich facial animation data and algorithms that can synthesize speech animations with emotional expressions at the same time. In addition, majority of the models are deterministic, meaning given the same audio input, they produce the same output motion. We argue that emotions and non-determinism are crucial to generate diverse and emotionally-rich facial animations. In this paper, we propose ProbTalk3D a non-deterministic neural network approach for emotion controllable speech-driven 3D facial animation synthesis using a two-stage VQ-VAE model and an emotionally rich facial animation dataset 3DMEAD. We provide an extensive comparative analysis of our model against the recent 3D facial animation synthesis approaches, by evaluating the results objectively, qualitatively, and with a perceptual user study. We highlight several objective metrics that are more suitable for evaluating stochastic outputs and use both in-the-wild and ground truth data for subjective evaluation. To our knowledge, that is the first non-deterministic 3D facial animation synthesis method incorporating a rich emotion dataset and emotion control with emotion labels and intensity levels. Our evaluation demonstrates that the proposed model achieves superior performance compared to state-of-the-art emotion-controlled, deterministic and non-deterministic models. We recommend watching the supplementary video for quality judgement. The entire codebase is publicly available (<a class="link-external link-https" href="https://github.com/uuembodiedsocialai/ProbTalk3D/" rel="external noopener nofollow">this https URL</a>).

Allo-AVA: A Large-Scale Multimodal Conversational AI Dataset for Allocentric Avatar Gesture Animation

Large Body Language Models

From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations

Capture, Learning, and Synthesis of 3D Speaking Styles

Avatar Fingerprinting for Authorized Use of Synthetic Talking-Head Videos

Learning Speech-driven 3D Conversational Gestures from Video

High-fidelity facial and speech animation for VR HMDs

AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding

MM-Conv: A Multi-modal Conversational Dataset for Virtual Humans

GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars

SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark

BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis

MMHead: Towards Fine-grained Multi-modal 3D Facial Animation

Attention-Based VR Facial Animation with Visual Mouth Camera Guidance for Immersive Telepresence Avatars

RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars

ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE

Expressive Talking Avatars

ECAvatar: 3D Avatar Facial Animation with Controllable Identity and Emotion

One Shot, One Talk: Whole-body Talking Avatar from a Single Image

Universal Facial Encoding of Codec Avatars from VR Headsets