Abstract:Automatically generating videos in which synthesized speech is synchronized with lip movements in a talking head has great potential in many human-computer interaction scenarios. In this paper, we present an automatic method to generate synchronized speech and talking-head videos on the basis of text and a single face image of an arbitrary person as input. In contrast to previous text-driven talking head generation methods, which can only synthesize the voice of a specific person, the proposed method is capable of synthesizing speech for any person that is inaccessible in the training stage. Specifically, the proposed method decomposes the generation of synchronized speech and talking head videos into two stages, i.e., a text-to-speech (TTS) stage and a speech-driven talking head generation stage. The proposed TTS module is a face-conditioned multi-speaker TTS model that gets the speaker identity information from face images instead of speech, which allows us to synthesize a personalized voice on the basis of the input face image. To generate the talking head videos from the face images, a facial landmark-based method that can predict both lip movements and head rotations is proposed. Extensive experiments demonstrate that the proposed method is able to generate synchronized speech and talking head videos for arbitrary persons and non-persons. Synthesized speech shows consistency with the given face regarding to the synthesized voice's timbre and one's appearance in the image, and the proposed landmark-based talking head method outperforms the state-of-the-art landmark-based method on generating natural talking head videos.

Speaker Independent and Multilingual/Mixlingual Speech-Driven Talking Head Generation Using Phonetic Posteriorgrams

Audio-driven Talking Face Video Generation with Natural Head Pose

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior

GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer

Listen, Disentangle, and Control: Controllable Speech-Driven Talking Head Generation

Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Experts

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation

MakeItTalk: Speaker-Aware Talking-Head Animation

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models

Talking Head Generation with Audio and Speech Related Facial Action Units

PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features

FaceFormer: Speech-Driven 3D Facial Animation with Transformers

GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting

Learning Landmarks Motion from Speech for Speaker-Agnostic 3D Talking Heads Generation

Spatially and Temporally Optimized Audio‐Driven Talking Face Generation

A Multimodal Approach of Generating 3D Human-Like Talking Agent.