Abstract:Modern text-to-speech (TTS) systems are able to generate audio that sounds almost as natural as human speech. However, the bar of developing high-quality TTS systems remains high since a sizable set of studio-quality pairs is usually required. Compared to commercial data used to develop state-of-the-art systems, publicly available data are usually worse in terms of both quality and size. Audio generated by TTS systems trained on publicly available data tends to not only sound less natural, but also exhibits more background noise. In this work, we aim to lower TTS systems' reliance on high-quality data by providing them the textual knowledge extracted by deep pre-trained language models during training. In particular, we investigate the use of BERT to assist the training of Tacotron-2, a state of the art TTS consisting of an encoder and an attention-based decoder. BERT representations learned from large amounts of unlabeled text data are shown to contain very rich semantic and syntactic information about the input text, and have potential to be leveraged by a TTS system to compensate the lack of high-quality data. We incorporate BERT as a parallel branch to the Tacotron-2 encoder with its own attention head. For an input text, it is simultaneously passed into BERT and the Tacotron-2 encoder. The representations extracted by the two branches are concatenated and then fed to the decoder. As a preliminary study, although we have not found incorporating BERT into Tacotron-2 generates more natural or cleaner speech at a human-perceivable level, we observe improvements in other aspects such as the model is being significantly better at knowing when to stop decoding such that there is much less babbling at the end of the synthesized audio and faster convergence during training.

The Tacotron2-based IPA-to-Speech speech synthesis system.

ViSPer: A Multilingual TTS Approach Based on VITS Using Deep Feature Loss

A Preliminary Study on Deep Learning-based Chinese Text to Taiwanese Speech Synthesis System

DOP-Tacotron: a Fast Chinese TTS System with Local-based Attention

Bidirectional Decoding Tacotron for Attention Based Neural Speech Synthesis

Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

A Novel Method for Mandarin Speech Synthesis by Inserting Prosodic Structure Prediction into Tacotron2.

IMPROVING NATURALNESS AND CONTROLLABILITY OF SEQUENCE-TO-SEQUENCE SPEECH SYNTHESIS BY LEARNING LOCAL PROSODY REPRESENTATIONS

DIA-TTS: Deep-Inherited Attention-Based Text-to-Speech Synthesizer

AI Enhancements for Linguistic E-Learning System

Grapheme-to-phoneme conversion for Chinese text-to-speech

Knowledge-based Linguistic Encoding for End-to-End Mandarin Text-to-Speech Synthesis

Zero-Shot Text-to-Speech as Golden Speech Generator: A Systematic Framework and its Applicability in Automatic Pronunciation Assessment

A Chinese Text-to-Speech System

Neural Speech Synthesis with Transformer Network.

WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss

Revisiting IPA-based Cross-lingual Text-to-speech

Articulatory Phonetics Informed Controllable Expressive Speech Synthesis

PPPR: Portable Plug-in Prompt Refiner for Text to Audio Generation

NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

Pronunciation Dictionary-Free Multilingual Speech Synthesis Using Learned Phonetic Representations.