Abstract:Custom voice, a specific text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech data. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions that could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we use two acoustic encoders to extract an utterance-level vector and a sequence of phoneme-level vectors from the target speech during training; in inference, we extract the utterance-level vector from a reference speech and use an acoustic predictor to predict the phoneme-level vectors. 2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adaptation. We pre-train the source TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g., 20 sentences, about 1 minute speech. Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K specific parameters for each speaker, which demonstrates its effectiveness for custom voice. Audio samples are available at <a class="link-external link-https" href="https://speechresearch.github.io/adaspeech/" rel="external noopener nofollow">this https URL</a>.

Adaptive Text to Speech for Spontaneous Style

AdaptiveFormer: A Few-shot Speaker Adaptive Speech Synthesis Model Based on FastSpeech2

AdaSpeech: Adaptive Text to Speech for Custom Voice

Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models

Adaspeech 2: Adaptive Text to Speech with Untranscribed Data

AS-Speech: Adaptive Style for Speech Synthesis

AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios

SponTTS: modeling and transferring spontaneous style for TTS

Expressive TTS Driven by Natural Language Prompts Using Few Human Annotations

Text-aware and Context-aware Expressive Audiobook Speech Synthesis

StyleSpeech: Parameter-efficient Fine Tuning for Pre-trained Controllable Text-to-Speech

Prosody-controllable spontaneous TTS with neural HMMs

Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation

ArtSpeech: Adaptive Text-to-Speech Synthesis with Articulatory Representations

InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt

USAT: A Universal Speaker-Adaptive Text-to-Speech Approach

Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis

Towards Spontaneous Style Modeling with Semi-supervised Pre-training for Conversational Text-to-Speech Synthesis

StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis

Emotionally Situated Text-to-Speech Synthesis in User-Agent Conversation

StyleFusion TTS: Multimodal Style-control and Enhanced Feature Fusion for Zero-shot Text-to-speech Synthesis