Abstract:In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style, merely based on a few seconds of audio prompt and a simple textual style description prompt. Prior zero-shot TTS models and controllable TTS models either could only mimic the speaker's voice without further control and adjustment capabilities or were unrelated to speaker-specific voice generation. Therefore, ControlSpeech focuses on a more challenging new task-a TTS system with controllable timbre, content, and style at the same time. ControlSpeech takes speech prompts, content prompts, and style prompts as inputs and utilizes bidirectional attention and mask-based parallel decoding to capture corresponding codec representations in a discrete decoupling codec space. Moreover, we discovered the issue of text style controllability in a many-to-many mapping fashion and proposed the Style Mixture Semantic Density (SMSD) model to resolve this problem. SMSD module which is based on Gaussian mixture density networks, is designed to enhance the fine-grained partitioning and sampling capabilities of style semantic information and generate speech with more diverse styles. In terms of experiments, we make available a controllable model toolkit called ControlToolkit with a new style controllable dataset, some replicated baseline models and propose new metrics to evaluate both the control capability and the quality of generated audio in ControlSpeech. The relevant ablation studies validate the necessity of each component in ControlSpeech is necessary. We hope that ControlSpeech can establish the next foundation paradigm of controllable speech synthesis. The relevant code and demo are available at <a class="link-external link-https" href="https://github.com/jishengpeng/ControlSpeech" rel="external noopener nofollow">this https URL</a> .

The THU-HCSI Multi-Speaker Multi-Lingual Few-Shot Voice Cloning System for LIMMITS'24 Challenge

A multi-speaker multi-lingual voice cloning system based on vits2 for limmits 2024 challenge

Multi-Speaker Multi-Lingual VQTTS System for LIMMITS 2023 Challenge

The Multi-Speaker Multi-Style Voice Cloning Challenge 2021

The huya multi-speaker and multi-style speech synthesis system for m2voc challenge 2020

UNET-TTS: Improving Unseen Speaker and Style Transfer in One-Shot Voice Cloning

INVESTIGATING ON INCORPORATING PRETRAINED AND LEARNABLE SPEAKER REPRESENTATIONS FOR MULTI-SPEAKER MULTI-STYLE TEXT-TO-SPEECH

Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers

U-Style: Cascading U-nets with Multi-level Speaker and Style Modeling for Zero-Shot Voice Cloning

A real-time voice cloning system with multiple algorithms for speech quality improvement

ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec

Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning

MRMI-TTS: Multi-reference audios and Mutual Information Driven Zero-shot Voice cloning

ZMM-TTS: Zero-shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-supervised Discrete Speech Representations

Multispeech: Multi-speaker text to speech with transformer

LeanSpeech: the Microsoft Lightweight Speech Synthesis System for Limmits Challenge 2023

OpenVoice: Versatile Instant Voice Cloning

The Codec Language Model-based Zero-Shot Spontaneous Style TTS System for CoVoC Challenge 2024

MuLanTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2023

Optimizing feature fusion for improved zero-shot adaptation in text-to-speech synthesis

HAM-TTS: Hierarchical Acoustic Modeling for Token-Based Zero-Shot Text-to-Speech with Model and Data Scaling