Abstract:Audio-driven emotional 3D face animation aims to generate emotionally expressive talking heads with synchronized lip movements. However, previous research has often overlooked the influence of diverse emotions on facial expressions or proved unsuitable for driving MetaHuman models. In response to this deficiency, we introduce EmoFace, a novel audio-driven methodology for creating facial animations with vivid emotional dynamics. Our approach can generate facial expressions with multiple emotions, and has the ability to generate random yet natural blinks and eye movements, while maintaining accurate lip synchronization. We propose independent speech encoders and emotion encoders to learn the relationship between audio, emotion and corresponding facial controller rigs, and finally map into the sequence of controller values. Additionally, we introduce two post-processing techniques dedicated to enhancing the authenticity of the animation, particularly in blinks and eye movements. Furthermore, recognizing the scarcity of emotional audio-visual data suitable for MetaHuman model manipulation, we contribute an emotional audio-visual dataset and derive control parameters for each frames. Our proposed methodology can be applied in producing dialogues animations of non-playable characters (NPCs) in video games, and driving avatars in virtual reality environments. Our further quantitative and qualitative experiments, as well as an user study comparing with existing researches show that our approach demonstrates superior results in driving 3D facial models. The code and sample data are available at <a class="link-external link-https" href="https://github.com/SJTU-Lucy/EmoFace" rel="external noopener nofollow">this https URL</a>.

EmoPlayer: A Media Player for Video Clips with Affective Annotations.

A color bar based affective annotation method for media player

An Eye Tracking Study: Positive Emotional Interface Design Facilitates Learning Outcomes in Multimedia Learning?

EmoG: Supporting the Sketching of Emotional Expressions for Storyboarding.

EmoEdit: Evoking Emotions through Image Manipulation

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

A Novel Affective Visualization System for Videos Based on Acoustic and Visual Features

Multimedia content analysis for emotional characterization of music video clips

AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues

EmoFace: Audio-driven Emotional 3D Face Animation

Identifying affective levels on music video via completing the missing modality

Video emotion analysis enhanced by recognizing emotion in video comments

Empathetic video clip experience through timely multimodal interaction

Inter-Brain EEG Feature Extraction and Analysis for Continuous Implicit Emotion Tagging During Video Watching

Video indexing and recommendation based on affective analysis of viewers.

User Intent-Oriented Video Qoe with Emotion Detection Networking

Implicit Media Tagging and Affect Prediction from video of spontaneous facial expressions, recorded with depth camera

EmoWear: Exploring Emotional Teasers for Voice Message Interaction on Smartwatches

<i>EmotionCues</i>: Emotion-Oriented Visual Summarization of Classroom Videos

Hybrid video emotional tagging using users’ EEG and video content

Content-Based Video Emotion Tagging Augmented by Users’ Multiple Physiological Responses