Abstract:This paper presents a realistic visual speech synthesis based on the hybrid concatenation method. Unlike previous methods based on phoneme level unit selection or hidden Markov model (HMM), etc., the hybrid concatenation method uses a frame level-based unit selection method combined with a fused HMM, and is able to generate more expressive and stable facial animations. The fused HMM can be used to explicitly model the loose synchronization of tightly coupled streams, with much better results than a normal HMM for audiovisual mapping. After fused HMM is created, facial animation is generated via the unit selection method at the frame level by using the fused HMM output probabilities. To accelerate the computing efficiency of the unit selection on a large corpus, this paper also proposes a two-layer Viterbi search method in which only the subsets that have been selected in the first layer are further checked in the second layer. Using this idea, the system has been successfully integrated into real-time applications. Furthermore, the paper also proposes a mapping method to generate emotional facial expressions from neutral facial expressions based on Gaussian mixture models (GMMs). Final experiments prove that the method described can output synthesized facial parameters with high quality. Compared with other audiovisual mapping methods, our method has better performance with respect to expressiveness, stability, and system running speed.

Improved visual speech synthesis method of parameter control

Articulatory Control of HMM-based Parametric Speech Synthesis Driven by Phonetic Knowledge

Improving Speech Intelligibility of Older Hearing Impaired Adults by Promoting Consonant-Vowel Ratio

New synthesis method based on LMA vocal tract model

Improving F0 prediction using bidirectional associative memories and syllable-level F0 features for HMM-based Mandarin speech synthesis

Evaluation of a Method for Vowel-Specific Voice Source Control of an Electrolarynx Using Visual Information

Improved unit selection speech synthesis method utilizing subjective evaluation results on synthetic speech

Audiovisual Synthesis of Exaggerated Speech for Corrective Feedback in Computer-Assisted Pronunciation Training

Improving HMM Based Speech Synthesis by Reducing Over-Smoothing Problems

Voiced/unvoiced Decision Algorithm for HMM-based Speech Synthesis

Evaluation of parameter generation using high order dynamic features and long span windows for HMM based speech synthesis

Improving Accuracy in Imitating and Reading Aloud via Speech Visualization Technology

Modifying Spectral Envelope to Synthetically Adjust Voice Quality and Articulation Parameters for Emotional Speech Synthesis

A New Acoustic Modeling of Inter-Syllable Context-Dependent Units for Putonghua Continuous Speech Recognition

Formant-Controlled HMM-Based Speech Synthesis.

Vowel Creation by Articulatory Control in HMM-based Parametric Speech Synthesis

Feature-Space Transform Tying in Unified Acoustic-Articulatory Modelling for Articulatory Control of HMM-Based Speech Synthesis.

Integrating Articulatory Features into HMM-Based Parametric Speech Synthesis

An Improved Cross-Language Model Adaptation Method for Speech Synthesis

Improving Deep Neural Network Based Speech Synthesis Through Contextual Feature Parametrization and Multi-Task Learning

Realistic Visual Speech Synthesis Based on Hybrid Concatenation Method