Abstract:Expressive synthetic speech is essential for many human-computer interaction and audio broadcast scenarios, and thus synthesizing expressive speech has attracted much attention in recent years. Previous methods performed the expressive speech synthesis either with explicit labels or with a fixed-length style embedding extracted from reference audio, both of which can only learn an average style and thus ignores the multi-scale nature of speech prosody. In this paper, we propose MsEmoTTS, a multi-scale emotional speech synthesis framework, to model the emotion from different levels. Specifically, the proposed method is a typical attention-based sequence-to-sequence model and with proposed three modules, including global-level emotion presenting module (GM), utterance-level emotion presenting module (UM), and local-level emotion presenting module (LM), to model the global emotion category, utterance-level emotion variation, and syllable-level emotion strength, respectively. In addition to modeling the emotion from different levels, the proposed method also allows us to synthesize emotional speech in different ways, i.e., transferring the emotion from reference audio, predicting the emotion from input text, and controlling the emotion strength manually. Extensive experiments conducted on a Chinese emotional speech corpus demonstrate that the proposed method outperforms the compared reference audio-based and text-based emotional speech synthesis methods on the emotion transfer speech synthesis and text-based emotion prediction speech synthesis respectively. Besides, the experiments also show that the proposed method can control the emotion expressions flexibly. Detailed analysis shows the effectiveness of each module and the good design of the proposed method.

Emotional Statistical Parametric Speech Synthesis Using LSTM-RNNs

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

A Comparison of Expressive Speech Synthesis Approaches based on Neural Network

Emotional speaker recognition based on similar neighbor phenomenon

A Preliminary Study on GMM Weight Transformation for Emotional Speaker Recognition

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

HMM-based emotional speech synthesis using average emotion model

Speaker-Independent Speech Emotion Recognition Based On Cnn-Blstm And Multiple Svms

Emphatic Speech Generation with Conditioned Input Layer and Bidirectional LSTMS for Expressive Speech Synthesis.

Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices

Prosody Analysis And Modeling For Emotional Speech Synthesis

Emotional Audio-Visual Speech Synthesis Based on PAD

Spontaneous Speech Emotion Recognition Using Multiscale Deep Convolutional LSTM

MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis

Language Model-Based Emotion Prediction Methods for Emotional Speech Synthesis Systems

Speech Synthesis with Mixed Emotions

Expressive Speech Driven Talking Avatar Synthesis with DBLSTM Using Limited Amount of Emotional Bimodal Data

Speech Emotion Classification Using Attention-Based LSTM

Exemplar-Based Emotive Speech Synthesis

Model architectures to extrapolate emotional expressions in DNN-based text-to-speech

Hybrid Deep Neural Network--Hidden Markov Model (DNN-HMM) Based Speech Emotion Recognition