Abstract:Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audio-visual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.

DeepTalk: Vocal Style Encoding for Speaker Recognition and Speech Synthesis

Vocal Style Factorization for Effective Speaker Recognition in Affective Scenarios

Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation

Deep Learning based Multilingual Speech Synthesis using Multi Feature Fusion Methods

Style-Label-Free: Cross-Speaker Style Transfer by Quantized VAE and Speaker-wise Normalization in Speech Synthesis

Make-A-Voice: Unified Voice Synthesis With Discrete Representation

Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style

Deep Learning-based F0 Synthesis for Speaker Anonymization

Learning Latent Representations for Style Control and Transfer in End-to-end Speech Synthesis

U-Style: Cascading U-nets with Multi-level Speaker and Style Modeling for Zero-Shot Voice Cloning

Deep Speech Synthesis from MRI-Based Articulatory Representations

Is Style All You Need? Dependencies Between Emotion and GST-based Speaker Recognition

Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Model

StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation

Deep Voice: Real-time Neural Text-to-Speech

Self-supervised Context-aware Style Representation for Expressive Speech Synthesis

Enhancing audio quality for expressive Neural Text-to-Speech

Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion

Deep Speaker: an End-to-End Neural Speaker Embedding System

Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis

VECL-TTS: Voice identity and Emotional style controllable Cross-Lingual Text-to-Speech