Abstract:Talking head generation is a significant research topic that still faces numerous challenges. Previous works often adopt generative adversarial networks or regression models, which are plagued by generation quality and average facial shape problem. Although diffusion models show impressive generative ability, their exploration in talking head generation remains unsatisfactory. This is because they either solely use the diffusion model to obtain an intermediate representation and then employ another pre-trained renderer, or they overlook the feature decoupling of complex facial details, such as expressions, head poses and appearance textures. Therefore, we propose a Facial Decoupled Diffusion model for Talking head generation called FD2Talk, which fully leverages the advantages of diffusion models and decouples the complex facial details through multi-stages. Specifically, we separate facial details into motion and appearance. In the initial phase, we design the Diffusion Transformer to accurately predict motion coefficients from raw audio. These motions are highly decoupled from appearance, making them easier for the network to learn compared to high-dimensional RGB images. Subsequently, in the second phase, we encode the reference image to capture appearance textures. The predicted facial and head motions and encoded appearance then serve as the conditions for the Diffusion UNet, guiding the frame generation. Benefiting from decoupling facial details and fully leveraging diffusion models, extensive experiments substantiate that our approach excels in enhancing image quality and generating more accurate and diverse results compared to previous state-of-the-art methods.

Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters

Multimodal-driven Talking Face Generation, Face Swapping, Diffusion Model

Multimodal-driven Talking Face Generation via a Unified Diffusion-based Generator

Audio-driven Talking Face Video Generation with Natural Head Pose

Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation

Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model

FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model

EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model

Conversational Co-Speech Gesture Generation via Modeling Dialog Intention, Emotion, and Context with Diffusion Models

DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer

DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models

Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness

Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation

Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation

DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation