Abstract:Speech-driven 3D facial animation is a challenging cross-modal task that has attracted growing research interest. During speaking activities, the mouth displays strong motions, while the other facial regions typically demonstrate comparatively weak activity levels. Existing approaches often simplify the process by directly mapping single-level speech features to the entire facial animation, which overlook the differences in facial activity intensity leading to overly smoothed facial movements. In this study, we propose a novel framework, CorrTalk, which effectively establishes the temporal correlation between hierarchical speech features and facial activities of different intensities across distinct regions. A novel facial activity intensity metric is defined to distinguish between strong and weak facial activity, obtained by computing the short-time Fourier transform of facial vertex displacements. Based on the variances in facial activity, we propose a dual-branch decoding framework to synchronously synthesize strong and weak facial activity, which guarantees wider intensity facial animation synthesis. Furthermore, a weighted hierarchical feature encoder is proposed to establish temporal correlation between hierarchical speech features and facial activity at different intensities, which ensures lip-sync and plausible facial expressions. Extensive qualitatively and quantitatively experiments as well as a user study indicate that our CorrTalk outperforms existing state-of-the-art methods. The source code and supplementary video are publicly available at: <a class="link-external link-https" href="https://zjchu.github.io/projects/CorrTalk/" rel="external noopener nofollow">this https URL</a>

Rhythm Correlation of Speech Synthesis System

Automatic Prosody Quality Evaluation of Mandarin Speech

Prosody Variation: Application to Automatic Prosody Evaluation of Mandarin Speech

SR-TTS: a rhyme-based end-to-end speech synthesis system

Rhythm-controllable Attention with High Robustness for Long Sentence Speech Synthesis

TTSDS -- Text-to-Speech Distribution Score

Objective Evaluation Methods for Chinese Text-To-Speech Systems

Into-TTS : Intonation Template Based Prosody Control System

Total Quality Evaluation of Speech Synthesis Systems.

Mandarin Speech Synthesis Based on Pitch Synchronous Time-Frequency Interpolation

A novel unit selection method for concatenation speech system using similarity measure

Multilingual Speech Evaluation: Case Studies on English, Malay and Tamil

An Optimized Neural Network Based Prosody Model of Chinese Speech Synthesis System

Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech

Comparison of Pronunciation Scores in Spoken Language Learning System

CorrTalk: Correlation Between Hierarchical Speech and Facial Activity Variances for 3D Animation

Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding

HMM based speech synthesis with Global Variance Training method

Toward Expressive Singing Voice Correction: On Perceptual Validity of Evaluation Metrics for Vocal Melody Extraction

Study on the Correlation between Objective Evaluations and Subjective Speech Quality and Intelligibility

Mandarin Stress Analysis And Prediction For Speech Synthesis