Abstract:We present a novel approach for synthesizing 3D facial motions from audio sequences using key motion embeddings. Despite recent advancements in data-driven techniques, accurately mapping between audio signals and 3D facial meshes remains challenging. Direct regression of the entire sequence often leads to over-smoothed results due to the ill-posed nature of the problem. To this end, we propose a progressive learning mechanism that generates 3D facial animations by introducing key motion capture to decrease cross-modal mapping uncertainty and learning complexity. Concretely, our method integrates linguistic and data-driven priors through two modules: the linguistic-based key motion acquisition and the cross-modal motion completion. The former identifies key motions and learns the associated 3D facial expressions, ensuring accurate lip-speech synchronization. The latter extends key motions into a full sequence of 3D talking faces guided by audio features, improving temporal coherence and audio-visual consistency. Extensive experimental comparisons against existing state-of-the-art methods demonstrate the superiority of our approach in generating more vivid and consistent talking face animations. Consistent enhancements in results through the integration of our proposed learning scheme with existing methods underscore the efficacy of our approach. Our code and weights will be at the project website: \url{<a class="link-external link-https" href="https://github.com/ffxzh/KMTalk" rel="external noopener nofollow">this https URL</a>}.

Audio To Visual Signal Mappings With Hmm

Dynamic Audio-Visual Mapping using Fused Hidden Markov Model Inversion Method

Face Animation Based on Large Audiovisual Database

Mining Audio/Visual Database For Speech Driven Face Animation

Realistic Visual Speech Synthesis Based on Hybrid Concatenation Method

Expressive Face Animation Synthesis Based on Dynamic Mapping Method

Real-time Lip Synchronization Based on Hidden Markov Models

KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding

Data Mining and Speech Driven Face Animation

Evaluation Of Linear Regression For Speaker Adaptation In Hmm-Based Articulatory Movements Estimation

Learning Audio-Driven Viseme Dynamics for 3D Face Animation

Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

Audio-driven facial animation by joint end-to-end learning of pose and emotion

Audio-Driven 3D Facial Animation from In-the-Wild Videos

Acoustic to Articulatory Mapping with Deep Neural Network

Dynamic mapping method based speech driven face animation system

A Multimodal Approach of Generating 3D Human-Like Talking Agent.

DreamHead: Learning Spatial-Temporal Correspondence via Hierarchical Diffusion for Audio-driven Talking Head Synthesis

MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentions

The Hidden Dance of Phonemes and Visage: Unveiling the Enigmatic Link between Phonemes and Facial Features