Abstract:While considerable progress has been made in achieving accurate lip synchronization for 3D speech-driven talking face generation, the task of incorporating expressive facial detail synthesis aligned with the speaker's speaking status remains challenging. Existing efforts either focus on learning a dynamic talking head pose synchronized with speech rhythm or aim for stylized facial movements guided by external reference such as emotional labels or reference video clips. The former works often yield coarse alignment, neglecting the emotional nuances present in the audio content while the latter studies lead to unnatural applications, requiring manual style source selection by users. Our goal is to directly leverage the inherent style information conveyed by human speech for generating an expressive talking face that aligns with the speaking status. In this paper, we propose AVI-Talking, an Audio-Visual Instruction system for expressive Talking face generation. This system harnesses the robust contextual reasoning and hallucination capability offered by Large Language Models (LLMs) to instruct the realistic synthesis of 3D talking faces. Instead of directly learning facial movements from human speech, our two-stage strategy involves the LLMs first comprehending audio information and generating instructions implying expressive facial details seamlessly corresponding to the speech. Subsequently, a diffusion-based generative network executes these instructions. This two-stage process, coupled with the incorporation of LLMs, enhances model interpretability and provides users with flexibility to comprehend instructions and specify desired operations or modifications. Specifically, given a speech clip, we first employ a Q-Former for contrastive alignment the speech features with visual instructions, which is then projected to input text embedding of LLMs. It functions as a prompting strategy, prompting LLMs to generate plausible visual instructions that encompass diverse facial details. In order to use these predicted instructions, a language-guided talking face generation system with disentangled latent space is delicately derived, where the speech content related lip movements and emotion correlated facial expressions are separately represented in speech content space and content irrelevant space. Additionally, we introduce a contrastive instruction-style alignment and diffusion technique within the content-irrelevant space to fully exploit the talking prior network for diverse instruction-following synthesis. Extensive experiments showcase the effectiveness of our approach in producing vivid talking faces with expressive facial movements and consistent emotional status.

Learn2Talk: 3D Talking Face Learns from 2D Talking Face

Audio-driven Talking Face Video Generation with Natural Head Pose

SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

Meta Talk: Learning To Data-Efficiently Generate Audio-Driven Lip-Synchronized Talking Face With High Definition

AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation

Audio-Driven 3D Facial Animation from In-the-Wild Videos

Pose-Aware 3D Talking Face Synthesis using Geometry-guided Audio-Vertices Attention

FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning

CLTalk: Speech-Driven 3D Facial Animation with Contrastive Learning

KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding

DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animation

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

End-to-end Learning for 3D Facial Animation from Raw Waveforms of Speech

Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation

SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation.

Talking Faces: Audio-to-Video Face Generation

Invariant Motion Representation Learning for 3D Talking Face Synthesis.

VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior