Abstract:Different people have different facial expressions while speaking emotionally. A realistic facial animation system should consider such identity-specific speaking styles and facial idiosyncrasies to achieve high-degree of naturalness and plausibility. Existing approaches to personalized speech-driven 3D facial animation either use one-hot identity labels or rely-on person specific models which limit their scalability. We present a personalized speech-driven expressive 3D facial animation synthesis framework that models identity specific facial motion as latent representations (called as styles), and synthesizes novel animations given a speech input with the target style for various emotion categories. Our framework is trained in an end-to-end fashion and has a non-autoregressive encoder-decoder architecture with three main components: expression encoder, speech encoder and expression decoder. Since, expressive facial motion includes both identity-specific style and speech-related content information; expression encoder first disentangles facial motion sequences into style and content representations, respectively. Then, both of the speech encoder and the expression decoders input the extracted style information to update transformer layer weights during training phase. Our speech encoder also extracts speech phoneme label and duration information to achieve better synchrony within the non-autoregressive synthesis mechanism more effectively. Through detailed experiments, we demonstrate that our approach produces temporally coherent facial expressions from input speech while preserving the speaking styles of the target identities.

Individual 3D Face Synthesis Based on Orthogonal Photos and Speech-Driven Facial Animation

GPU-Accelerated Parallel Realistic 3D Facial Expression Synthesis

Audio-driven Talking Face Video Generation with Natural Head Pose

Personalized facial animation based on 3d model fitting from two orthogonal face images

Individual facial image synthesis system for a virtual human

Speech-Driven 3D Face Animation with Composite and Regional Facial Movements

Quick Customization of Particular Human Face and Muscle-Driven Expression Animation

Audio-Driven 3D Facial Animation from In-the-Wild Videos

Three-dimensional Animated Face Modeling from Stereo Video

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

3D Realistic Talking Face Co-Driven by Text and Speech

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion

Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Experts

Personalized 3-D facial expression synthesis based on landmark constraint.

Personalized Speech-driven Expressive 3D Facial Animation Synthesis with Style Control

3D Facial Animation from Chinese Text.

3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing

Facial Animation System for Embedded Application

Automatic Face Modeling and Synthesis Based on Image Pairs