Abstract:Expressive text-to-speech (TTS) aims to synthesize different speaking style speech according to human's demands. Nowadays, there are two common ways to control speaking styles: (1) Pre-defining a group of speaking style and using categorical index to denote different speaking style. However, there are limitations in the diversity of expressiveness, as these models can only generate the pre-defined styles. (2) Using reference speech as style input, which results in a problem that the extracted style information is not intuitive or interpretable. In this study, we attempt to use natural language as style prompt to control the styles in the synthetic speech, e.g., "Sigh tone in full of sad mood with some helpless feeling". Considering that there is no existing TTS corpus which is proper to benchmark this novel task, we first construct a speech corpus, whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an expressive TTS model, named as InstructTTS, which is novel in the sense of following aspects: (1) We fully take the advantage of self-supervised learning and cross-modal metric learning, and propose a novel three-stage training procedure to obtain a robust sentence embedding model, which can effectively capture semantic information from the style prompts and control the speaking style in the generated speech. (2) We propose to model acoustic features in discrete latent space and train a novel discrete diffusion probabilistic model to generate vector-quantized (VQ) acoustic tokens rather than the commonly-used mel spectrogram. (3) We jointly apply mutual information (MI) estimation and minimization during acoustic model training to minimize style-speaker and style-content MI, avoiding possible content and speaker information leakage from the style prompt.

Modeling the Acoustic Correlates of Dialog Act for Expressive Chinese TTS Synthesis

MODELLING THE GLOBAL ACOUSTIC CORRELATES OF EXPRESSIVITY FOR CHINESE TEXT-TO-SPEECH SYNTHESIS

Modeling the Acoustic Correlates of Expre Expressive Text-to-Spe

Modeling the Expressivity of Input Text Semantics for Chinese Text-to-Speech Synthesis in a Spoken Dialog System

Text-driven Visual Prosody Generation for Embodied Conversational Agents

Modeling Prosody Patterns for Chinese Expressive Text-to-speech Synthesis

Articulatory Phonetics Informed Controllable Expressive Speech Synthesis

Toward Synthesizing Expressive Mandarin Speech

Prosody Analysis And Modeling For Emotional Speech Synthesis

Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling

Hierarchical Stress Modeling and Generation in Mandarin for Expressive Text-to-Speech

InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt

Improving Prosody with Linguistic and Bert Derived Features in Multi-Speaker Based Mandarin Chinese Neural TTS

Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech

Prosody Modelling with Pre-trained Cross-utterance Representations for Improved Speech Synthesis

FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis

An Optimized Neural Network Based Prosody Model of Chinese Speech Synthesis System

An Expressive Mandarin Speech Corpus

Learning Prosodic Patterns for Mandarin Speech Synthesis

Head Movement Synthesis Based on Semantic and Prosodic Features for a Chinese Expressive Avatar

EE-TTS: Emphatic Expressive TTS with Linguistic Information