Abstract:By representing speaker characteristic as a single fixed-length vector extracted solely from speech, we can train a neural multi-speaker speech synthesis model by conditioning the model on those vectors. This model can also be adapted to unseen speakers regardless of whether the transcript of adaptation data is available or not. However, this setup restricts the speaker component to just a single bias vector, which in turn limits the performance of adaptation process. In this study, we propose a novel speech synthesis model, which can be adapted to unseen speakers by fine-tuning part of or all of the network using either transcribed or untranscribed speech. Our methodology essentially consists of two steps: first, we split the conventional acoustic model into a speaker-independent (SI) linguistic encoder and a speaker-adaptive (SA) acoustic decoder; second, we train an auxiliary acoustic encoder that can be used as a substitute for the linguistic encoder whenever linguistic features are unobtainable. The results of objective and subjective evaluations show that adaptation using either transcribed or untranscribed speech with our methodology achieved a reasonable level of performance with an extremely limited amount of data and greatly improved performance with more data. Surprisingly, adaptation with untranscribed speech surpassed the transcribed counterpart in the subjective test, which reveals the limitations of the conventional acoustic model and hints at potential directions for improvements.

Speaker adaptation using maximum likelihood model interpolation

A Speaker Adaptation Algorithm Based on Matrix Linear Interpolation

Linguistic tree based maximum likelihood model interpolation

Agmma: A Novel Incremental Adaptation Method And Its Application To Speaker Recognition

Rapid Speaker Adaptation Using Multi-Stream Structural Maximum Likelihood Eigenspace Mapping

MAP-based Speaker Adaptation in Speech Synthesis

Discriminative Speaker Adaptation with Eigenvoices

Speaker Normalization Training and Adaptation for Speech Recognition

Speech Recognition Using Speaker Adaptation by System Parameter Transformation.

Phoneme Dependent Speaker Embedding And Model Factorization For Multi-Speaker Speech Synthesis And Adaptation

Speaker Adaptation and Adaptive Training for Jointly Optimised Tandem Systems.

A New Subspace Based Speaker Adaptation Method

Speaker adaptation based on combination of MAP estimation and weighted neighbor regression

Speaker Adaptation with MAP Estimation and Weighted Neighbor Regression

Improving Online Incremental Speaker Adaptation with Eigen Feature Space MLLR.

Linear Networks Based Speaker Adaptation for Speech Synthesis

Speaker Adaptation for Telephony Data Using Speaker Clustering

Interpolation adaptation algorithm based on gaussian similarity analysis

A Unified Speaker Adaptation Method for Speech Synthesis using Transcribed and Untranscribed Speech with Backpropagation

A Novel speaker Adaptation Method based on Map and NNLR

Model Adaptation for HMM-Based Speech Synthesis under Minimum Generation Error Criterion