Abstract:Conversational Text-to-Speech (CTTS) aims to accurately express an utterance with the appropriate style within a conversational setting, which attracts more attention nowadays. While recognizing the significance of the CTTS task, prior studies have not thoroughly investigated speech emphasis expression, which is essential for conveying the underlying intention and attitude in human-machine interaction scenarios, due to the scarcity of conversational emphasis datasets and the difficulty in context understanding. In this paper, we propose a novel Emphasis Rendering scheme for the CTTS model, termed ER-CTTS, that includes two main components: 1) we simultaneously take into account textual and acoustic contexts, with both global and local semantic modeling to understand the conversation context comprehensively; 2) we deeply integrate multi-modal and multi-scale context to learn the influence of context on the emphasis expression of the current utterance. Finally, the inferred emphasis feature is fed into the neural speech synthesizer to generate conversational speech. To address data scarcity, we create emphasis intensity annotations on the existing conversational dataset (DailyTalk). Both objective and subjective evaluations suggest that our model outperforms the baseline models in emphasis rendering within a conversational setting. The code and audio samples are available at <a class="link-external link-https" href="https://github.com/CodeStoreTTS/ER-CTTS" rel="external noopener nofollow">this https URL</a>.

Learning Cross-Lingual Knowledge With Multilingual Blstm For Emphasis Detection With Limited Training Data

Cross-Lingual Text Image Recognition Via Multi-Task Sequence to Sequence Learning.

Attention Bidirectional LSTM Networks Based Mime Speech Recognition Using Semg Data

LEARNING CROSS-LINGUAL INFORMATION WITH MULTILINGUAL BLSTM FOR SPEECH SYNTHESIS OF LOW-RESOURCE LANGUAGES

Improved BLSTM RNN Based Accent Speech Recognition Using Multi-task Learning and Accent Embeddings

Emphasis Detection for Voice Dialogue Applications Using Multi-channel Convolutional Bidirectional Long Short-Term Memory Network

Emphatic Speech Generation with Conditioned Input Layer and Bidirectional LSTMS for Expressive Speech Synthesis.

Generating emphatic speech with hidden Markov model for expressive speech synthesis

Learning Contextual Representation with Convolution Bank and Multi-head Self-attention for Speech Emphasis Detection.

Inferring Emphasis for Real Voice Data: an Attentive Multimodal Neural Network Approach.

Improving Accented Mandarin Speech Recognition by Using Recurrent Neural Network Based Language Model Adaptation

Hierarchical English Emphatic Speech Synthesis Based on HMM with Limited Training Data.

Multilingual Recurrent Neural Networks with Residual Learning for Low-Resource Speech Recognition.

Recurrent Neural Network Based Language Model Adaptation for Accent Mandarin Speech.

Improving Blstm Rnn Based Mandarin Speech Recognition Using Accent Dependent Bottleneck Features

Multi-task Recurrent Model for True Multilingual Speech Recognition

Biomedical Named Entity Recognition via A Hybrid Neural Network Model

Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modeling

Cross-Lingual Self-training to Learn Multilingual Representation for Low-Resource Speech Recognition

Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision

Bidirectional LSTM with Extended Input Context