Abstract:This paper proposes a speech synthesis method based on unsupervisedly-learned fine-grained style representations, named word-level style variations (WSVs), in order to improve the naturalness of synthetic speech. The whole model contains a WSV extractor and a WSV predictor. The WSV extractor is jointly trained with a sequence-to-sequence (Seq2seq) synthesizer and learns a WSV vector from the mel-spectrogram of each prosodic word in the training set by extending the global style token (GST) framework. In contrast to GST weights which describe the global styles of utterances, WSVs operate at word-level and are expected to describe local style properties, such as stresses. Besides, Gumbel softmax is adopted and the extracted WSVs are close to one-hot vectors which facilitate the subsequent prediction task. The WSV predictor is a deterministic model which generates the sequence of WSV vectors from input text using an autoregressive LSTM network. In addition to phonetic information, e.g., phoneme sequences, Bidirectional Encoder Representation from Transformers (BERT) model is employed by the predictor to obtain the semantic descriptions of input text for better predicting the latent speech representation, i.e., WSVs. The WSV predictor is trained by considering both the accuracy of WSV prediction and the distortion of mel-spectrograms recovered from the predicted WSVs. Experimental results show that our proposed method can achieve better naturalness of synthetic speech than baseline Tacotron2, text-predicted global style token (TP-GST) and BERT-Tacotron2 models.

The Importance Weighted Autoencoder in End-to-End Speech Synthesis

Learning Latent Representations for Style Control and Transfer in End-to-end Speech Synthesis

End-to-End Emotional Speech Synthesis Method Based on Conditional Variational Autoencoder

Extracting and Predicting Word-Level Style Variations for Speech Synthesis

Joint and Adversarial Training with ASR for Expressive Speech Synthesis

End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Training

Learning Hierarchical Representations for Expressive Speaking Style in End-to-End Speech Synthesis

Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis

Unsupervised Learning for Multi-Style Speech Synthesis with Limited Data

Innovative Speaker-Adaptive Style Transfer VAE-WadaIN for Enhanced Voice Conversion in Intelligent Speech Processing

StyleSpeech: Self-supervised Style Enhancing with VQ-VAE-based Pre-training for Expressive Audiobook Speech Synthesis

AS-Speech: Adaptive Style for Speech Synthesis

Text-aware and Context-aware Expressive Audiobook Speech Synthesis

Self-supervised Context-aware Style Representation for Expressive Speech Synthesis

Using Multiple Reference Audios and Style Embedding Constraints for Speech Synthesis

Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder

Style-Label-Free: Cross-Speaker Style Transfer by Quantized VAE and Speaker-wise Normalization in Speech Synthesis

Rapid Style Adaptation Using Residual Error Embedding for Expressive Speech Synthesis.

Speaking Style Compensation on Synthetic Audio for Robust Keyword Spotting.

Towards Multi-Scale Style Control for Expressive Speech Synthesis

BI-LEVEL STYLE AND PROSODY DECOUPLING MODELING FOR PERSONALIZED END-TO-END SPEECH SYNTHESIS