Abstract:We present VQTalker, a Vector Quantization-based framework for multilingual talking head generation that addresses the challenges of lip synchronization and natural motion across diverse languages. Our approach is grounded in the phonetic principle that human speech comprises a finite set of distinct sound units (phonemes) and corresponding visual articulations (visemes), which often share commonalities across languages. We introduce a facial motion tokenizer based on Group Residual Finite Scalar Quantization (GRFSQ), which creates a discretized representation of facial features. This method enables comprehensive capture of facial movements while improving generalization to multiple languages, even with limited training data. Building on this quantized representation, we implement a coarse-to-fine motion generation process that progressively refines facial animations. Extensive experiments demonstrate that VQTalker achieves state-of-the-art performance in both video-driven and speech-driven scenarios, particularly in multilingual settings. Notably, our method achieves high-quality results at a resolution of 512*512 pixels while maintaining a lower bitrate of approximately 11 kbps. Our work opens new possibilities for cross-lingual talking face generation. Synthetic results can be viewed at <a class="link-external link-https" href="https://x-lance.github.io/VQTalker" rel="external noopener nofollow">this https URL</a>.

Phoneme Embedding and its Application to Speech Driven Talking Avatar Synthesis

Enhanced Double-Carrier Word Embedding Via Phonetics and Writing

Expressive Speech Driven Talking Avatar Synthesis with DBLSTM Using Limited Amount of Emotional Bimodal Data

Low Level Descriptors Based DBLSTM Bottleneck Feature for Speech Driven Talking Avatar

VQTalker: Towards Multilingual Talking Avatars through Facial Motion Tokenization

Word Embedding For Recurrent Neural Network Based Tts Synthesis

Improving Speech Enhancement with Phonetic Embedding Features

Introducing Phonetic Information to Speaker Embedding for Speaker Verification

The Parameterized Phoneme Identity Feature As a Continuous Real-Valued Vector for Neural Network Based Speech Synthesis.

Multilingual and Crosslingual Speech Recognition Using Phonological-Vector Based Phone Embeddings

Acoustic Word Embeddings for End-to-End Speech Synthesis

Speech-driven Head Motion Synthesis Using Neural Networks

Phoneme Dependent Speaker Embedding And Model Factorization For Multi-Speaker Speech Synthesis And Adaptation

A real-time speech driven talking avatar based on deep neural network.

On the Usage of Phonetic Information for Text-Independent Speaker Embedding Extraction.

Acoustic to Articulatory Mapping with Deep Neural Network

Improve Bilingual TTS Using Dynamic Language and Phonology Embedding

Speech Rhythm-Based Speaker Embeddings Extraction from Phonemes and Phoneme Duration for Multi-Speaker Speech Synthesis

Phonetic-aware speaker embedding for far-field speaker verification

Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers

Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis