Abstract:To be widely adopted, 3D facial avatars must be animated easily, realistically, and directly from speech signals. While the best recent methods generate 3D animations that are synchronized with the input audio, they largely ignore the impact of emotions on facial expressions. Realistic facial animation requires lip-sync together with the natural expression of emotion. To that end, we propose EMOTE (Expressive Model Optimized for Talking with Emotion), which generates 3D talking-head avatars that maintain lip-sync from speech while enabling explicit control over the expression of emotion. To achieve this, we supervise EMOTE with decoupled losses for speech (i.e., lip-sync) and emotion. These losses are based on two key observations: (1) deformations of the face due to speech are spatially localized around the mouth and have high temporal frequency, whereas (2) facial expressions may deform the whole face and occur over longer intervals. Thus, we train EMOTE with a per-frame lip-reading loss to preserve the speech-dependent content, while supervising emotion at the sequence level. Furthermore, we employ a content-emotion exchange mechanism in order to supervise different emotions on the same audio, while maintaining the lip motion synchronized with the speech. To employ deep perceptual losses without getting undesirable artifacts, we devise a motion prior in the form of a temporal VAE. Due to the absence of high-quality aligned emotional 3D face datasets with speech, EMOTE is trained with 3D pseudo-ground-truth extracted from an emotional video dataset (i.e., MEAD). Extensive qualitative and perceptual evaluations demonstrate that EMOTE produces speech-driven facial animations with better lip-sync than state-of-the-art methods trained on the same data, while offering additional, high-quality emotional control.

Emotional Voice Puppetry

EmoFace: Audio-driven Emotional 3D Face Animation

Voicing Your Emotion: Integrating Emotion and Identity in Cross-Modal 3D Facial Animations

Emotional Speech-Driven Animation with Content-Emotion Disentanglement

Emotional Talking Agent: System and Evaluation

Emotional Audio-Visual Speech Synthesis Based on PAD

Expressive Speech-driven Facial Animation with controllable emotions

FusionCraft: Fusing Emotion and Identity in Cross-Modal 3D Facial Animation

EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation

Multimodal emotion estimation and emotional synthesize for interaction virtual agent

Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait

Audio-Driven Emotional Video Portraits

Facial Expression Synthesis Using PAD Emotional Parameters for a Chinese Expressive Avatar

Expressive Talking Avatars

EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

Enhancing Emotional Induction in Virtual Reality: Innovative Fusion of Speech and Realistic Non-Verbal Emotional Expression

Real-time Speech-Driven Animation of Expressive Talking Faces.

EmoFace: Emotion-Content Disentangled Speech-Driven 3D Talking Face with Mesh Attention

Facial Expression Synthesis Based on Emotion Dimensions for Affective Talking Avatar

Real-time Synthesis of Chinese Visual Speech and Facial Expressions Using MPEG-4 FAP Features in a Three-Dimensional Avatar

Visual Speech Emotion Conversion Using Deep Learning for 3D Talking Head