Abstract:Expressive synthetic speech is essential for many human-computer interaction and audio broadcast scenarios, and thus synthesizing expressive speech has attracted much attention in recent years. Previous methods performed the expressive speech synthesis either with explicit labels or with a fixed-length style embedding extracted from reference audio, both of which can only learn an average style and thus ignores the multi-scale nature of speech prosody. In this paper, we propose MsEmoTTS, a multi-scale emotional speech synthesis framework, to model the emotion from different levels. Specifically, the proposed method is a typical attention-based sequence-to-sequence model and with proposed three modules, including global-level emotion presenting module (GM), utterance-level emotion presenting module (UM), and local-level emotion presenting module (LM), to model the global emotion category, utterance-level emotion variation, and syllable-level emotion strength, respectively. In addition to modeling the emotion from different levels, the proposed method also allows us to synthesize emotional speech in different ways, i.e., transferring the emotion from reference audio, predicting the emotion from input text, and controlling the emotion strength manually. Extensive experiments conducted on a Chinese emotional speech corpus demonstrate that the proposed method outperforms the compared reference audio-based and text-based emotional speech synthesis methods on the emotion transfer speech synthesis and text-based emotion prediction speech synthesis respectively. Besides, the experiments also show that the proposed method can control the emotion expressions flexibly. Detailed analysis shows the effectiveness of each module and the good design of the proposed method.

Emotional Speech Synthesis Based on the Modification of Prosody Parameters and Spectral Envelope

Modifying Spectral Envelope to Synthetically Adjust Voice Quality and Articulation Parameters for Emotional Speech Synthesis

Prosody Analysis And Modeling For Emotional Speech Synthesis

Prosody Conversion from Neutral Speech to Emotional Speech.

Speech Synthesis with Mixed Emotions

Toward Synthesizing Expressive Mandarin Speech

Emotional Audio-Visual Speech Synthesis Based on PAD

EmoPro: A Prompt Selection Strategy for Emotional Expression in LM-based Speech Synthesis

Emotional Speech Synthesis Based on Improved Codebook Mapping Voice Conversion

Nonlinear emotional prosody generation and annotation

EMPHASIS: An Emotional Phoneme-based Acoustic Model for Speech Synthesis System

MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis

Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis

EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech

Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions

Two-Stage Prosody Prediction For Emotional Text-To-Speech Synthesis

Emotional Speech Generation by Using Statistic Prosody Conversion Methods

Towards Realistic Emotional Voice Conversion using Controllable Emotional Intensity

Emotional Talking Agent: System and Evaluation

MODELLING THE GLOBAL ACOUSTIC CORRELATES OF EXPRESSIVITY FOR CHINESE TEXT-TO-SPEECH SYNTHESIS

Emotional speech synthesis with rich and granularized control