Abstract:While previous audio-driven talking head generation (THG) methods generate head poses from driving audio, the generated poses or lips cannot match the audio well or are not editable. In this study, we propose \textbf{PoseTalk}, a THG system that can freely generate lip-synchronized talking head videos with free head poses conditioned on text prompts and audio. The core insight of our method is using head pose to connect visual, linguistic, and audio signals. First, we propose to generate poses from both audio and text prompts, where the audio offers short-term variations and rhythm correspondence of the head movements and the text prompts describe the long-term semantics of head motions. To achieve this goal, we devise a Pose Latent Diffusion (PLD) model to generate motion latent from text prompts and audio cues in a pose latent space. Second, we observe a loss-imbalance problem: the loss for the lip region contributes less than 4\% of the total reconstruction loss caused by both pose and lip, making optimization lean towards head movements rather than lip shapes. To address this issue, we propose a refinement-based learning strategy to synthesize natural talking videos using two cascaded networks, i.e., CoarseNet, and RefineNet. The CoarseNet estimates coarse motions to produce animated images in novel poses and the RefineNet focuses on learning finer lip motions by progressively estimating lip motions from low-to-high resolutions, yielding improved lip-synchronization performance. Experiments demonstrate our pose prediction strategy achieves better pose diversity and realness compared to text-only or audio-only, and our video generator model outperforms state-of-the-art methods in synthesizing talking videos with natural head motions. Project: <a class="link-external link-https" href="https://junleen.github.io/projects/posetalk" rel="external noopener nofollow">this https URL</a>.

ListenFormer: Responsive Listening Head Generation with Non-autoregressive Transformers

Video-driven speaker-listener generation based on Transformer and neural renderer

CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation

AudioVSR: Enhancing Video Speech Recognition with Audio Data

Responsive Listening Head Generation: A Benchmark Dataset and Baseline

Can Language Models Learn to Listen?

Audio-driven Talking Face Video Generation with Natural Head Pose

MFR-Net: Multi-faceted Responsive Listening Head Generation via Denoising Diffusion Model

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

Hierarchical Semantic Perceptual Listener Head Video Generation: A High-performance Pipeline

FaceFormer: Speech-Driven 3D Facial Animation with Transformers

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

Listen, Disentangle, and Control: Controllable Speech-Driven Talking Head Generation

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

FoleyGen: Visually-Guided Audio Generation

PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Interactive Conversational Head Generation

Predicting Personalized Head Movement From Short Video and Speech Signal

Emotional Listener Portrait: Neural Listener Head Generation with Emotion