CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens

Zhihao Du,Qian Chen,Shiliang Zhang,Kai Hu,Heng Lu,Yexin Yang,Hangrui Hu,Siqi Zheng,Yue Gu,Ziyang Ma,Zhifu Gao,Zhijie Yan

2024-07-09

Abstract:Recent years have witnessed a trend that large language model (LLM) based text-to-speech (TTS) emerges into the mainstream due to their high naturalness and zero-shot capacity. In this paradigm, speech signals are discretized into token sequences, which are modeled by an LLM with text as prompts and reconstructed by a token-based vocoder to waveforms. Obviously, speech tokens play a critical role in LLM-based TTS models. Current speech tokens are learned in an unsupervised manner, which lacks explicit semantic information and alignment to the text. In this paper, we propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis. Experimental results show that supervised semantic tokens significantly outperform existing unsupervised tokens in terms of content consistency and speaker similarity for zero-shot voice cloning. Moreover, we find that utilizing large-scale data further improves the synthesis performance, indicating the scalable capacity of CosyVoice. To the best of our knowledge, this is the first attempt to involve supervised speech tokens into TTS models.

Sound,Artificial Intelligence,Audio and Speech Processing

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the following issues:

1. **Improving the quality of zero-shot speech synthesis**: Currently, text-to-speech (TTS) systems based on large language models (LLMs) often use unsupervised methods to obtain speech tokens when generating highly natural and zero-shot capable speech signals. This approach lacks explicit semantic information and good alignment with the text. This paper proposes a supervised semantic token-based method, extracting speech tokens through a multilingual speech recognition model and applying them to the TTS system to improve content consistency and speaker similarity.

2. **Proposing an efficient and scalable TTS synthesizer**: The paper introduces CosyVoice, a zero-shot TTS system that combines LLMs for text-to-token generation and a conditional flow matching model for token-to-speech synthesis. Compared to traditional diffusion models, this method is faster during training and inference and does not require additional phonemizers and forced alignment tools.

3. **Enhancing the controllability of generated speech**: By integrating instruction fine-tuning, CosyVoice gains the ability to control speaker identity, speaking style, and fine-grained paralinguistic features, enabling more flexible and high-quality speech synthesis.

Through these improvements, the paper demonstrates the superior performance of CosyVoice in zero-shot speech cloning tasks and validates its scalability on large-scale datasets.