Abstract:Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audio-visual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.

Audio-Driven Stylized Gesture Generation with Flow-Based Model.

DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models

DiT-Gesture: A Speech-Only Approach to Stylized Gesture Generation

TalkingFlow: Talking Facial Landmark Generation with Multi-Scale Normalizing Flow Network

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach

Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

Audio-Driven Co-Speech Gesture Video Generation

Autoregressive Stylized Motion Synthesis with Generative Flow

Analyzing Input and Output Representations for Speech-Driven Gesture Generation

EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation

Speech gesture generation from the trimodal context of text, audio, and speaker identity

Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation

Audio2Gestures: Generating Diverse Gestures From Audio

ExpGest: Expressive Speaker Generation Using Diffusion Model and Hybrid Audio-Text Guidance

Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness

Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents

Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model

Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style