Abstract:Multi-speaker text-to-speech synthesis involves generating unique speech patterns for individual speakers based on reference waveforms and input sequences of graphemes or phonemes. Various deep neural networks are trained for this task using a large amount of speech data recorded from a specific speaker to generate audio in their voice. The model requires a large dataset to retrain itself and learn about a new speaker not seen during training. This process is expensive in terms of time and resources. Thus, a key requirement of such techniques is to reduce time and resource consumption. In this paper, a multi-speaker text-to-speech synthesis using a generalized end-to-end loss function is developed, capable of generating speech in real-time for a given speech reference from a user and a text string as input. This method considers the speaker's characteristics in the generated speech using the speech reference of their voice. The proposed method also assesses the effect on spontaneity and fluency in the generated language, corresponding to the speaker encoder, using the mean opinion score (MOS). However, a speaker encoder is trained with varying hours of the audio dataset, and it observes the effect on the produced speech. Furthermore, an extensive analysis is performed on the impact of the training dataset on the speaker encoder, corresponding to the generated speech, and various speaker encoder models for the speaker verification task. Based on loss function and Equal Error Rate (EER), advanced GRU is selected for generalized end-to-end loss function. The speaker verification regression test represents that the projected prototype can generate language, which the regression algorithm is able to distinguish into two sets: male and female while second test shows the above technique is able to distinguish speaker embeddings separately in clusters showing each speaker is uniquely identified. In terms of results, our proposed model achieved a MOS of 4.02 when trained on 'Train Clean 100', 3.74 on 'Train-clean-360', and 3.25 on 'Train-clean-500'. The MOS test juxtaposes our method with prior models, demonstrating its superior performance. Conclusively, a cross-similarity matrix offers a visual representation of the similarity and disparity between utterances, underscoring the model's robustness and efficacy.

Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings

Content-Dependent Fine-Grained Speaker Embedding for Zero-Shot Speaker Adaptation in Text-to-Speech Synthesis

Adapting End-to-End Neural Speaker Verification to New Languages and Recording Conditions with Adversarial Training

Deep Speaker: an End-to-End Neural Speaker Embedding System

A Unified Speaker Adaptation Method for Speech Synthesis using Transcribed and Untranscribed Speech with Backpropagation

Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representations

Zero-shot personalized lip-to-speech synthesis with face image based voice control

Speaker-Smoothed kNN Speaker Adaptation for End-to-End ASR

Deep neural network-based speaker embeddings for end-to-end speaker verification

Joint speaker encoder and neural back-end model for fully end-to-end automatic speaker verification with multiple enrollment utterances

Phoneme Dependent Speaker Embedding And Model Factorization For Multi-Speaker Speech Synthesis And Adaptation

AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios

Dynamic Speaker Representations Adjustment and Decoder Factorization for Speaker Adaptation in End-to-End Speech Synthesis

Multi speaker text-to-speech synthesis using generalized end-to-end loss function

Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis

Deep Voice 2: Multi-Speaker Neural Text-to-Speech

ZMM-TTS: Zero-shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-supervised Discrete Speech Representations

An analysis on the effects of speaker embedding choice in non auto-regressive TTS

USAT: A Universal Speaker-Adaptive Text-to-Speech Approach

Training Multi-Speaker Neural Text-to-Speech Systems using Speaker-Imbalanced Speech Corpora

Post-Training Embedding Alignment for Decoupling Enrollment and Runtime Speaker Recognition Models