Abstract:Conventional text-to-speech (TTS) research has predominantly focused on enhancing the quality of synthesized speech for speakers in the training dataset. The challenge of synthesizing lifelike speech for unseen, out-of-dataset speakers, especially those with limited reference data, remains a significant and unresolved problem. While zero-shot or few-shot speaker-adaptive TTS approaches have been explored, they have many limitations. Zero-shot approaches tend to suffer from insufficient generalization performance to reproduce the voice of speakers with heavy accents. While few-shot methods can reproduce highly varying accents, they bring a significant storage burden and the risk of overfitting and catastrophic forgetting. In addition, prior approaches only provide either zero-shot or few-shot adaptation, constraining their utility across varied real-world scenarios with different demands. Besides, most current evaluations of speaker-adaptive TTS are conducted only on datasets of native speakers, inadvertently neglecting a vast portion of non-native speakers with diverse accents. Our proposed framework unifies both zero-shot and few-shot speaker adaptation strategies, which we term as "instant" and "fine-grained" adaptations based on their merits. To alleviate the insufficient generalization performance observed in zero-shot speaker adaptation, we designed two innovative discriminators and introduced a memory mechanism for the speech decoder. To prevent catastrophic forgetting and reduce storage implications for few-shot speaker adaptation, we designed two adapters and a unique adaptation procedure.

Spoken Content and Voice Factorization for Few-Shot Speaker Adaptation

Phoneme Dependent Speaker Embedding And Model Factorization For Multi-Speaker Speech Synthesis And Adaptation

Prosody and voice factorization for few-shot speaker adaptation in the challenge m2voc 2021

Dynamic Speaker Representations Adjustment and Decoder Factorization for Speaker Adaptation in End-to-End Speech Synthesis

Content-Dependent Fine-Grained Speaker Embedding for Zero-Shot Speaker Adaptation in Text-to-Speech Synthesis

A Unified Speaker Adaptation Method for Speech Synthesis using Transcribed and Untranscribed Speech with Backpropagation

Speaker Adaption with Intuitive Prosodic Features for Statistical Parametric Speech Synthesis

USAT: A Universal Speaker-Adaptive Text-to-Speech Approach

A New Subspace Based Speaker Adaptation Method

Focusing on attention: prosody transfer and adaptative optimization strategy for multi-speaker end-to-end speech synthesis

Linear Networks Based Speaker Adaptation for Speech Synthesis

Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems

An Improved Cross-Language Model Adaptation Method for Speech Synthesis

AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios

Adaptive Text to Speech for Spontaneous Style

Cantonese neural speech synthesis from found newscasting video data and its speaker adaptation

ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptation

Label Transform Based Cross-Language Speaker Adaptation in Bilingual (Mandarin-English) TTS

Duration optimization of speaker adaptation in Mandarin TTS

Speech Recognition Using Speaker Adaptation by System Parameter Transformation.

Resource-Efficient Adaptation of Speech Foundation Models for Multi-Speaker ASR