Abstract:Speech synthesis (text to speech, TTS) and recognition (automatic speech recognition, ASR) are important speech tasks, and require a large amount of text and speech pairs for model training. However, there are more than 6,000 languages in the world and most languages are lack of speech training data, which poses significant challenges when building TTS and ASR systems for extremely low-resource languages. In this paper, we develop LRSpeech, a TTS and ASR system under the extremely low-resource setting, which can support rare languages with low data cost. LRSpeech consists of three key techniques: 1) pre-training on rich-resource languages and fine-tuning on low-resource languages; 2) dual transformation between TTS and ASR to iteratively boost the accuracy of each other; 3) knowledge distillation to customize the TTS model on a high-quality target-speaker voice and improve the ASR model on multiple voices. We conduct experiments on an experimental language (English) and a truly low-resource language (Lithuanian) to verify the effectiveness of LRSpeech. Experimental results show that LRSpeech 1) achieves high quality for TTS in terms of both intelligibility (more than 98% intelligibility rate) and naturalness (above 3.5 mean opinion score (MOS)) of the synthesized speech, which satisfy the requirements for industrial deployment, 2) achieves promising recognition accuracy for ASR, and 3) last but not least, uses extremely low-resource training data. We also conduct comprehensive analyses on LRSpeech with different amounts of data resources, and provide valuable insights and guidances for industrial deployment. We are currently deploying LRSpeech into a commercialized cloud speech service to support TTS on more rare languages.

Rapid development of a Latvian speech-to-text system

LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition

What shall we do with an hour of data? Speech recognition for the un- and under-served languages of Common Voice

Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASR

Zero Shot Text to Speech Augmentation for Automatic Speech Recognition on Low-Resource Accented Speech Corpora

Textless Speech-to-Speech Translation With Limited Parallel Data

Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models

Low-data? No problem: low-resource, language-agnostic conversational text-to-speech via F0-conditioned data augmentation

OLISIA: a Cascade System for Spoken Dialogue State Tracking

Low Latency ASR for Simultaneous Speech Translation

Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corpora

Open source platform for Estonian speech transcription

Neural Speech Synthesis for Estonian

Multi-Speaker Multi-Lingual VQTTS System for LIMMITS 2023 Challenge

MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech

Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach

Error-preserving Automatic Speech Recognition of Young English Learners' Language

Exploring Retraining-Free Speech Recognition for Intra-sentential Code-Switching

Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent

VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment

A multilingual training strategy for low resource Text to Speech