Abstract:In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style, merely based on a few seconds of audio prompt and a simple textual style description prompt. Prior zero-shot TTS models and controllable TTS models either could only mimic the speaker's voice without further control and adjustment capabilities or were unrelated to speaker-specific voice generation. Therefore, ControlSpeech focuses on a more challenging new task-a TTS system with controllable timbre, content, and style at the same time. ControlSpeech takes speech prompts, content prompts, and style prompts as inputs and utilizes bidirectional attention and mask-based parallel decoding to capture corresponding codec representations in a discrete decoupling codec space. Moreover, we discovered the issue of text style controllability in a many-to-many mapping fashion and proposed the Style Mixture Semantic Density (SMSD) model to resolve this problem. SMSD module which is based on Gaussian mixture density networks, is designed to enhance the fine-grained partitioning and sampling capabilities of style semantic information and generate speech with more diverse styles. In terms of experiments, we make available a controllable model toolkit called ControlToolkit with a new style controllable dataset, some replicated baseline models and propose new metrics to evaluate both the control capability and the quality of generated audio in ControlSpeech. The relevant ablation studies validate the necessity of each component in ControlSpeech is necessary. We hope that ControlSpeech can establish the next foundation paradigm of controllable speech synthesis. The relevant code and demo are available at <a class="link-external link-https" href="https://github.com/jishengpeng/ControlSpeech" rel="external noopener nofollow">this https URL</a> .

Controllable Prosody Generation With Partial Inputs

Prosodic Parameter Manipulation in TTS generated speech for Controlled Speech Generation

Prosody-TTS: An End-to-End Speech Synthesis System with Prosody Control

Prosody-controllable spontaneous TTS with neural HMMs

Towards Fine-Grained Prosody Control for Voice Conversion

Controllable Speaking Styles Using a Large Language Model

Hierarchical Prosody Modeling and Control in Non-Autoregressive Parallel Neural TTS

Robust and fine-grained prosody control of end-to-end speech synthesis

Into-TTS : Intonation Template Based Prosody Control System

Emotional Prosody Control for Speech Generation

Ensemble prosody prediction for expressive speech synthesis

Unsupervised word-level prosody tagging for controllable speech synthesis

IMPROVING NATURALNESS AND CONTROLLABILITY OF SEQUENCE-TO-SEQUENCE SPEECH SYNTHESIS BY LEARNING LOCAL PROSODY REPRESENTATIONS

A Novel Prosody Adaptation Method for Mandarin Concatenation-Based Text-to-speech System

Towards Controllable Speech Synthesis in the Era of Large Language Models: A Survey

Focusing on attention: prosody transfer and adaptative optimization strategy for multi-speaker end-to-end speech synthesis

Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit

Toward Invariant Functional Representations of Variable Surface Fundamental Frequency Contours: Synthesizing Speech Melody Via Model-Based Stochastic Learning

PRESENT: Zero-Shot Text-to-Prosody Control

Prosody Analysis And Modeling For Emotional Speech Synthesis

ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec