Abstract:Expressive text-to-speech (TTS) has become a hot research topic recently, mainly focusing on modeling prosody in speech. Prosody modeling has several challenges: 1) the extracted pitch used in previous prosody modeling works have inevitable errors, which hurts the prosody modeling; 2) different attributes of prosody (e.g., pitch, duration and energy) are dependent on each other and produce the natural prosody together; and 3) due to high variability of prosody and the limited amount of high-quality data for TTS training, the distribution of prosody cannot be fully shaped. To tackle these issues, we propose ProsoSpeech, which enhances the prosody using quantized latent vectors pre-trained on large-scale unpaired and low-quality text and speech data. Specifically, we first introduce a word-level prosody encoder, which quantizes the low-frequency band of the speech and compresses prosody attributes in the latent prosody vector (LPV). Then we introduce an LPV predictor, which predicts LPV given word sequence. We pre-train the LPV predictor on large-scale text and low-quality speech data and fine-tune it on the high-quality TTS dataset. Finally, our model can generate expressive speech conditioned on the predicted LPV. Experimental results show that ProsoSpeech can generate speech with richer prosody compared with baseline methods.

A New Prosodic Strength Calculation Method for Prosody Reduction Modeling

Modeling Prosody Patterns for Chinese Expressive Text-to-speech Synthesis

Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit

An Innovative Prosody Modeling Method for Chinese Speech Recognition

HIERARCHICAL PROSODY MODELING FOR NON-AUTOREGRESSIVE SPEECH SYNTHESIS

Prosody Model for Mandarin Text-to-Speech System

A Novel Prosody Adaptation Method for Mandarin Concatenation-Based Text-to-speech System

Prosody Modification on Mixed-language Speech Synthesis

Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech

Learning Prosodic Patterns for Mandarin Speech Synthesis

Prosody Analysis And Modeling For Emotional Speech Synthesis

Modeling prosody pattern of Chinese expressive speech and its application in personalized speech conversion

Prosodic Correlation Model in Text-to-Speech Synthesis

Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWP

Prosodic Modeling with Rich Syntactic Context in HMM-based Mandarin Speech Synthesis

ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech

Modeling the Acoustic Correlates of Expre Expressive Text-to-Spe

Automatic Prosody Annotation with Pre-Trained Text-Speech Model

Improving Prosody for Unseen Texts in Speech Synthesis by Utilizing Linguistic Information and Noisy Data

Ensemble prosody prediction for expressive speech synthesis

Mandarin dialog prosody model