Abstract:Audio-driven emotional 3D face animation aims to generate emotionally expressive talking heads with synchronized lip movements. However, previous research has often overlooked the influence of diverse emotions on facial expressions or proved unsuitable for driving MetaHuman models. In response to this deficiency, we introduce EmoFace, a novel audio-driven methodology for creating facial animations with vivid emotional dynamics. Our approach can generate facial expressions with multiple emotions, and has the ability to generate random yet natural blinks and eye movements, while maintaining accurate lip synchronization. We propose independent speech encoders and emotion encoders to learn the relationship between audio, emotion and corresponding facial controller rigs, and finally map into the sequence of controller values. Additionally, we introduce two post-processing techniques dedicated to enhancing the authenticity of the animation, particularly in blinks and eye movements. Furthermore, recognizing the scarcity of emotional audio-visual data suitable for MetaHuman model manipulation, we contribute an emotional audio-visual dataset and derive control parameters for each frames. Our proposed methodology can be applied in producing dialogues animations of non-playable characters (NPCs) in video games, and driving avatars in virtual reality environments. Our further quantitative and qualitative experiments, as well as an user study comparing with existing researches show that our approach demonstrates superior results in driving 3D facial models. The code and sample data are available at <a class="link-external link-https" href="https://github.com/SJTU-Lucy/EmoFace" rel="external noopener nofollow">this https URL</a>.

Expressive lip-synchronization facial animation system

Real-time Speech-Driven Animation of Expressive Talking Faces.

Facial Expression Animation Based on Physical Model

Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation

Expressive Face Animation Synthesis Based on Dynamic Mapping Method

Expressive facial animation synthesis by learning speech coarticulation and expression spaces

Expressive Speech-driven Facial Animation with controllable emotions

Lip Animation System for Visual Speech Synthesis in Uyghur Language

Real-time speech-driven lip synchronization

3D Facial Animation from Chinese Text.

Expressive 3D Facial Animation Generation Based on Local-to-Global Latent Diffusion

Lip syncing method for realistic expressive 3D face model

Facial Animation System for Embedded Application

EmoFace: Audio-driven Emotional 3D Face Animation

Audio-Driven Lips and Expression on 3D Human Face.

CSTalk: Correlation Supervised Speech-driven 3D Emotional Facial Animation Generation

A new expression database and facial animation

A Synthesis Method of Three-dimensional Facial Animation with Multiple Elements Blending Based on MPEG-4

Real-time Synthesis of Chinese Visual Speech and Facial Expressions Using MPEG-4 FAP Features in a Three-Dimensional Avatar

EMMN: Emotional Motion Memory Network for Audio-driven Emotional Talking Face Generation