Abstract:Expressive text-to-speech (TTS) aims to synthesize different speaking style speech according to human's demands. Nowadays, there are two common ways to control speaking styles: (1) Pre-defining a group of speaking style and using categorical index to denote different speaking style. However, there are limitations in the diversity of expressiveness, as these models can only generate the pre-defined styles. (2) Using reference speech as style input, which results in a problem that the extracted style information is not intuitive or interpretable. In this study, we attempt to use natural language as style prompt to control the styles in the synthetic speech, e.g., "Sigh tone in full of sad mood with some helpless feeling". Considering that there is no existing TTS corpus which is proper to benchmark this novel task, we first construct a speech corpus, whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an expressive TTS model, named as InstructTTS, which is novel in the sense of following aspects: (1) We fully take the advantage of self-supervised learning and cross-modal metric learning, and propose a novel three-stage training procedure to obtain a robust sentence embedding model, which can effectively capture semantic information from the style prompts and control the speaking style in the generated speech. (2) We propose to model acoustic features in discrete latent space and train a novel discrete diffusion probabilistic model to generate vector-quantized (VQ) acoustic tokens rather than the commonly-used mel spectrogram. (3) We jointly apply mutual information (MI) estimation and minimization during acoustic model training to minimize style-speaker and style-content MI, avoiding possible content and speaker information leakage from the style prompt.

Modeling the Acoustic Correlates of Expre Expressive Text-to-Spe

MODELLING THE GLOBAL ACOUSTIC CORRELATES OF EXPRESSIVITY FOR CHINESE TEXT-TO-SPEECH SYNTHESIS

Modeling the Expressivity of Input Text Semantics for Chinese Text-to-Speech Synthesis in a Spoken Dialog System

Modeling the Acoustic Correlates of Dialog Act for Expressive Chinese TTS Synthesis

Modeling Prosody Patterns for Chinese Expressive Text-to-speech Synthesis

Prosody Analysis And Modeling For Emotional Speech Synthesis

Prosody modeling for mandarin exclamatory speech

Modeling prosody pattern of Chinese expressive speech and its application in personalized speech conversion

Analysis and Modeling of Affective Audio Visual Speech Based on PAD Emotion Space

InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt

Emotional Audio-Visual Speech Synthesis Based on PAD

Prosody Modeling with 3D Visual Information for Expressive Video Dubbing

Emotion-Aware Prosodic Phrasing for Expressive Text-to-Speech

Head and Facial Gestures Synthesis Using PAD Model for an Expressive Talking Avatar

Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech

Head Movement Synthesis Based on Semantic and Prosodic Features for a Chinese Expressive Avatar

A New Prosodic Strength Calculation Method for Prosody Reduction Modeling

Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation

StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations

A Discourse-level Multi-scale Prosodic Model for Fine-grained Emotion Analysis

Nonlinear emotional prosody generation and annotation