Abstract:Audio-driven emotional 3D face animation aims to generate emotionally expressive talking heads with synchronized lip movements. However, previous research has often overlooked the influence of diverse emotions on facial expressions or proved unsuitable for driving MetaHuman models. In response to this deficiency, we introduce EmoFace, a novel audio-driven methodology for creating facial animations with vivid emotional dynamics. Our approach can generate facial expressions with multiple emotions, and has the ability to generate random yet natural blinks and eye movements, while maintaining accurate lip synchronization. We propose independent speech encoders and emotion encoders to learn the relationship between audio, emotion and corresponding facial controller rigs, and finally map into the sequence of controller values. Additionally, we introduce two post-processing techniques dedicated to enhancing the authenticity of the animation, particularly in blinks and eye movements. Furthermore, recognizing the scarcity of emotional audio-visual data suitable for MetaHuman model manipulation, we contribute an emotional audio-visual dataset and derive control parameters for each frames. Our proposed methodology can be applied in producing dialogues animations of non-playable characters (NPCs) in video games, and driving avatars in virtual reality environments. Our further quantitative and qualitative experiments, as well as an user study comparing with existing researches show that our approach demonstrates superior results in driving 3D facial models. The code and sample data are available at <a class="link-external link-https" href="https://github.com/SJTU-Lucy/EmoFace" rel="external noopener nofollow">this https URL</a>.

A Computational Model of Emotion for 3D Talking Heads

Emotional Chinese talking head system

EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Head

3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head

Voicing Your Emotion: Integrating Emotion and Identity in Cross-Modal 3D Facial Animations

A Computational Model of Emotion Based on Audio-Visual Stimuli Understanding and Personalized Regulation with Concurrency

Emotional Talking Agent: System and Evaluation

EmoFace: Audio-driven Emotional 3D Face Animation

Emotional States Based 3-D Fuzzy Atmosfield for Casual Communication Between Humans and Robots

Expression animation of interactive virtual humans based on HMM emotion model

Facial Expression Synthesis Based on Emotion Dimensions for Affective Talking Avatar

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

Construction of virtual assistant based on basic emotions theory

EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation

Happy Companion : A System of Multimodal Human-Computer Affective Interaction

Multimodal emotion estimation and emotional synthesize for interaction virtual agent

High-fidelity Generalized Emotional Talking Face Generation with Multi-modal Emotion Space Learning

Visual Speech Emotion Conversion Using Deep Learning for 3D Talking Head

FusionCraft: Fusing Emotion and Identity in Cross-Modal 3D Facial Animation

Emotional expression system based on expression intensity recognition and affective model

Affective transfer computing model based on attenuation emotion mechanism