Abstract:While previous audio-driven talking head generation (THG) methods generate head poses from driving audio, the generated poses or lips cannot match the audio well or are not editable. In this study, we propose \textbf{PoseTalk}, a THG system that can freely generate lip-synchronized talking head videos with free head poses conditioned on text prompts and audio. The core insight of our method is using head pose to connect visual, linguistic, and audio signals. First, we propose to generate poses from both audio and text prompts, where the audio offers short-term variations and rhythm correspondence of the head movements and the text prompts describe the long-term semantics of head motions. To achieve this goal, we devise a Pose Latent Diffusion (PLD) model to generate motion latent from text prompts and audio cues in a pose latent space. Second, we observe a loss-imbalance problem: the loss for the lip region contributes less than 4\% of the total reconstruction loss caused by both pose and lip, making optimization lean towards head movements rather than lip shapes. To address this issue, we propose a refinement-based learning strategy to synthesize natural talking videos using two cascaded networks, i.e., CoarseNet, and RefineNet. The CoarseNet estimates coarse motions to produce animated images in novel poses and the RefineNet focuses on learning finer lip motions by progressively estimating lip motions from low-to-high resolutions, yielding improved lip-synchronization performance. Experiments demonstrate our pose prediction strategy achieves better pose diversity and realness compared to text-only or audio-only, and our video generator model outperforms state-of-the-art methods in synthesizing talking videos with natural head motions. Project: <a class="link-external link-https" href="https://junleen.github.io/projects/posetalk" rel="external noopener nofollow">this https URL</a>.

Multi-Modal Driven Pose-Controllable Talking Head Generation

Multimodal-driven Talking Face Generation, Face Swapping, Diffusion Model

Audio-Semantic Enhanced Pose-Driven Talking Head Generation

Audio-driven Talking Face Video Generation with Natural Head Pose

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

High-Fidelity and Freely Controllable Talking Head Video Generation

Multimodal Inputs Driven Talking Face Generation With Spatial–Temporal Dependency

A Multimodal Approach of Generating 3D Human-Like Talking Agent.

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

Talking Head Generation Driven by Speech-Related Facial Action Units and Audio- Based on Multimodal Representation Fusion

PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation

StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads

Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation

Multimodal Learning for Temporally Coherent Talking Face Generation with Articulator Synergy

OPT: One-shot Pose-Controllable Talking Head Generation

3D Talking Face with Personalized Pose Dynamics

Listen, Disentangle, and Control: Controllable Speech-Driven Talking Head Generation

FONT: Flow-guided One-shot Talking Head Generation with Natural Head Motions

MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentions

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Controllable Talking Face Generation by Implicit Facial Keypoints Editing