Abstract:Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audio-visual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.

Hierarchical Feature Warping and Blending for Talking Head Animation

OAW-GAN: Occlusion-Aware Warping GAN for Unified Human Video Synthesis

High-Fidelity and Freely Controllable Talking Head Video Generation

Animating Through Warping An Efficient Method for High-Quality Facial Expression Animation

Style Transfer for 2D Talking Head Animation

Audio-driven Talking Face Video Generation with Natural Head Pose

MakeItTalk: Speaker-Aware Talking-Head Animation

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

Hierarchical facial expression animation by motion capture data

Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field

Pose-Aware 3D Talking Face Synthesis using Geometry-guided Audio-Vertices Attention

Generating Smooth and Facial-Details-Enhanced Talking Head Video: A Perspective of Pre and Post Processes

Hierarchical Coding for Talking-Head Video

Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style

AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding

Towards High-fidelity Head Blending with Chroma Keying for Industrial Applications

Speech4Mesh: Speech-Assisted Monocular 3D Facial Reconstruction for Speech-Driven 3D Facial Animation

3D-Aware Talking-Head Video Motion Transfer

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose