Meta Learning Text-to-Speech Synthesis in over 7000 Languages

Florian Lux,Sarina Meyer,Lyonel Behringer,Frank Zalkow,Phat Do,Matt Coler,Emanuël A. P. Habets,Ngoc Thang Vu

2024-06-10

Abstract:In this work, we take on the challenging task of building a single text-to-speech synthesis system that is capable of generating speech in over 7000 languages, many of which lack sufficient data for traditional TTS development. By leveraging a novel integration of massively multilingual pretraining and meta learning to approximate language representations, our approach enables zero-shot speech synthesis in languages without any available data. We validate our system's performance through objective measures and human evaluation across a diverse linguistic landscape. By releasing our code and models publicly, we aim to empower communities with limited linguistic resources and foster further innovation in the field of speech technology.

Computation and Language,Machine Learning,Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The paper aims to address the issues of multilingual Text-to-Speech (TTS) systems, particularly focusing on synthesizing low-resource languages among the over 7000 languages in the world. The main objectives include:

1. **Large-scale Multilingual TTS System**: Constructing a TTS system capable of handling 7212 languages, covering almost all spoken languages.
2. **Zero-shot Speech Synthesis**: Achieving zero-shot speech synthesis in languages with no available data by approximating language representations through meta-learning techniques.
3. **Cross-lingual Generalization**: Overcoming differences between languages, especially in speech features and language structures, using large-scale pre-training and meta-learning methods.
4. **High-quality Speech Generation**: Ensuring high-quality and controllable speech generation, achieving good synthesis results even for low-resource languages.

The research team collected and cleaned a large-scale multilingual dataset and designed a novel loss function (LESS) to constrain the structure of the language embedding space, thereby achieving the above objectives. Additionally, they employed a meta-learning approach to estimate representations of unseen languages, enabling the system to generate speech in the complete absence of data. The paper validates the performance of the system through extensive testing using objective metrics and human evaluations.