Abstract:Speaker adaptation, which involves cloning voices from unseen speakers in the Text-to-Speech task, has garnered significant interest due to its numerous applications in multi-media fields. Despite recent advancements, existing methods often struggle with inadequate speaker representation accuracy and overfitting, particularly in limited reference speeches scenarios. To address these challenges, we propose an Agile Speaker Representation Reinforcement Learning strategy to enhance speaker similarity in speaker adaptation tasks. ASRRL is the first work to apply reinforcement learning to improve the modeling accuracy of speaker embeddings in speaker adaptation, addressing the challenge of decoupling voice content and timbre. Our approach introduces two action strategies tailored to different reference speeches scenarios. In the single-sentence scenario, a knowledge-oriented optimal routine searching RL method is employed to expedite the exploration and retrieval of refinement information on the fringe of speaker representations. In the few-sentence scenario, we utilize a dynamic RL method to adaptively fuse reference speeches, enhancing the robustness and accuracy of speaker modeling. To achieve optimal results in the target domain, a multi-scale fusion scoring mechanism based reward model that evaluates speaker similarity, speech quality, and intelligibility across three dimensions is proposed, ensuring that improvements in speaker similarity do not compromise speech quality or intelligibility. The experimental results on the LibriTTS and VCTK datasets within mainstream TTS frameworks demonstrate the extensibility and generalization capabilities of the proposed ASRRL method. The results indicate that the ASRRL method significantly outperforms traditional fine-tuning approaches, achieving higher speaker similarity and better overall speech quality with limited reference speeches.

Implicit Enhancement of Target Speaker in Speaker-Adaptive ASR Through Efficient Joint Optimization

SQ-Whisper: Speaker-Querying based Whisper Model for Target-Speaker ASR

PI-Whisper: An Adaptive and Incremental ASR Framework for Diverse and Evolving Speaker Characteristics

Dynamic Speaker Representations Adjustment and Decoder Factorization for Speaker Adaptation in End-to-End Speech Synthesis

Phoneme Dependent Speaker Embedding And Model Factorization For Multi-Speaker Speech Synthesis And Adaptation

Improving End-to-End Single-Channel Multi-Talker Speech Recognition.

Speaker-Invariant Training Via Adversarial Learning.

Enhancing and Adversarial: Improve ASR with Speaker Labels

META-CAT: Speaker-Informed Speech Embeddings via Meta Information Concatenation for Multi-talker ASR

Speaker-aware target speaker enhancement by jointly learning with speaker embedding extraction

A Speech Enhancement Algorithm By Iterating Single- And Multi-Microphone Processing And Its Application To Robust Asr

Resource-Efficient Adaptation of Speech Foundation Models for Multi-Speaker ASR

End-to-End Dereverberation, Beamforming, and Speech Recognition in a Cocktail Party.

Integrated Adaptation with Multi-Factor Joint-Learning for Far-Field Speech Recognition

Extending Whisper with prompt tuning to target-speaker ASR

Speaker Adaptation on Articulation and Acoustics for Articulation-to-Speech Synthesis

Improving Multi-Speaker ASR With Overlap-Aware Encoding And Monotonic Attention.

Joint Feature Enhancement and Speaker Recognition with Multi-Objective Task-Oriented Network.

Multi-Channel Multi-Speaker ASR Using Target Speaker's Solo Segment

Speaker-Smoothed kNN Speaker Adaptation for End-to-End ASR

ASRRL-TTS: Agile Speaker Representation Reinforcement Learning for Text-to-Speech Speaker Adaptation