Abstract:Abstract Deep learning techniques are currently being applied in automated text-to-speech (TTS) systems, resulting in significant improvements in performance. However, these methods require large amounts of text-speech paired data for model training, and collecting this data is costly. Therefore, in this paper, we propose a single-speaker TTS system containing both a spectrogram prediction network and a neural vocoder for the target language, using only 30 min of target language text-speech paired data for training. We evaluate three approaches for training the spectrogram prediction models of our TTS system, which produce mel-spectrograms from the input phoneme sequence: (1) cross-lingual transfer learning, (2) data augmentation, and (3) a combination of the previous two methods. In the cross-lingual transfer learning method, we used two high-resource language datasets, English (24 h) and Japanese (10 h). We also used 30 min of target language data for training in all three approaches, and for generating the augmented data used for training in methods 2 and 3. We found that using both cross-lingual transfer learning and augmented data during training resulted in the most natural synthesized target speech output. We also compare single-speaker and multi-speaker training methods, using sequential and simultaneous training, respectively. The multi-speaker models were found to be more effective for constructing a single-speaker, low-resource TTS model. In addition, we trained two Parallel WaveGAN (PWG) neural vocoders, one using 13 h of our augmented data with 30 min of target language data and one using the entire 12 h of the original target language dataset. Our subjective AB preference test indicated that the neural vocoder trained with augmented data achieved almost the same perceived speech quality as the vocoder trained with the entire target language dataset. Overall, we found that our proposed TTS system consisting of a spectrogram prediction network and a PWG neural vocoder was able to achieve reasonable performance using only 30 min of target language training data. We also found that by using 3 h of target language data, for training the model and for generating augmented data, our proposed TTS model was able to achieve performance very similar to that of the baseline model, which was trained with 12 h of target language data.

Using Transfer Learning to Realize Low Resource Dungan Language Speech Synthesis

Using Speech Enhancement to Realize Speech Synthesis of Low-Resource Dungan Languages

Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

Effective Fine-tuning Method for Tibetan Low-resource Dialect Speech Recognition

Transformer Speech Synthesis for Tibetan Lhasa Dialect Based on Multilevel Granular Unit

Low-Resource Mongolian Speech Synthesis Based on Automatic Prosody Annotation

Low-Resource Language Discrimination toward Chinese Dialects with Transfer Learning and Data Augmentation

Lightweight Convolution-Based Chinese Speech Synthesis Method

Effective Training End-to-End ASR systems for Low-resource Lhasa Dialect of Tibetan Language

Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus

Improving Low-Resource Tibetan End-to-end ASR by Multilingual and Multilevel Unit Modeling

Towards High-Quality Neural TTS for Low-Resource Languages by Learning Compact Speech Representations

EM-TTS: Efficiently Trained Low-Resource Mongolian Lightweight Text-to-Speech

Transfer Learning, Style Control, and Speaker Reconstruction Loss for Zero-Shot Multilingual Multi-Speaker Text-to-Speech on Low-Resource Languages

Low Resource Malay Dialect Automatic Speech Recognition Modeling Using Transfer Learning from a Standard Malay Model

LEARNING CROSS-LINGUAL INFORMATION WITH MULTILINGUAL BLSTM FOR SPEECH SYNTHESIS OF LOW-RESOURCE LANGUAGES

Finer-grained Modeling Units-Based Meta-Learning for Low-resource Tibetan Speech Recognition

A Novel Method for Mandarin Speech Synthesis by Inserting Prosodic Structure Prediction into Tacotron2.

Text-to-speech system for low-resource language using cross-lingual transfer learning and data augmentation

A Preliminary Study on Deep Learning-based Chinese Text to Taiwanese Speech Synthesis System

VoiceTuner: Self-Supervised Pre-training and Efficient Fine-tuning for Voice Generation