Abstract:Modern text-to-speech (TTS) systems are able to generate audio that sounds almost as natural as human speech. However, the bar of developing high-quality TTS systems remains high since a sizable set of studio-quality pairs is usually required. Compared to commercial data used to develop state-of-the-art systems, publicly available data are usually worse in terms of both quality and size. Audio generated by TTS systems trained on publicly available data tends to not only sound less natural, but also exhibits more background noise. In this work, we aim to lower TTS systems' reliance on high-quality data by providing them the textual knowledge extracted by deep pre-trained language models during training. In particular, we investigate the use of BERT to assist the training of Tacotron-2, a state of the art TTS consisting of an encoder and an attention-based decoder. BERT representations learned from large amounts of unlabeled text data are shown to contain very rich semantic and syntactic information about the input text, and have potential to be leveraged by a TTS system to compensate the lack of high-quality data. We incorporate BERT as a parallel branch to the Tacotron-2 encoder with its own attention head. For an input text, it is simultaneously passed into BERT and the Tacotron-2 encoder. The representations extracted by the two branches are concatenated and then fed to the decoder. As a preliminary study, although we have not found incorporating BERT into Tacotron-2 generates more natural or cleaner speech at a human-perceivable level, we observe improvements in other aspects such as the model is being significantly better at knowing when to stop decoding such that there is much less babbling at the end of the synthesized audio and faster convergence during training.

Towards Efficiently Learning Monotonic Alignments for Attention-based End-to-End Speech Recognition

Improving Attention-Based End-to-End Speech Recognition by Monotonic Alignment Attention Matrix Reconstruction.

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Efficient Decoding Self-Attention for End-to-end Speech Synthesis

An improved hybrid CTC-Attention model for speech recognition

Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment

EfficientTTS 2: Variational End-to-End Text-to-Speech Synthesis and Voice Conversion

Initial investigation of an encoder-decoder end-to-end TTS framework using marginalization of monotonic hard latent alignments

One TTS Alignment To Rule Them All

Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

Improving Multi-Speaker ASR With Overlap-Aware Encoding And Monotonic Attention.

Improving End-to-End Single-Channel Multi-Talker Speech Recognition.

An efficient joint training model for monaural noisy-reverberant speech recognition

Efficient Monotonic Multihead Attention

Enhancing Monotonicity for Robust Autoregressive Transformer TTS

Optimizing Alignment of Speech and Language Latent Spaces for End-To-End Speech Recognition and Understanding.

Weak Alignment Supervision from Hybrid Model Improves End-to-end ASR

Attention-Constrained Inference for Robust Decoder-Only Text-to-Speech

A hybrid CTC+Attention model based on end-to-end framework for multilingual speech recognition

Online Hybrid CTC/Attention End-to-End Automatic Speech Recognition Architecture

Attention-based Transducer for Online Speech Recognition