Voice Style Cloning for Chinese Speech

Haiyun Wang,Ziqi Chen,Luoyi Yang

Abstract:In this project, we focus on training and evaluating an end-to-end text-to-speech synthesis system named Tacotron. Different from traditional text-to-speech model which requires engineers to train three modules including a frontend text analyzer, an acoustic module

Computer Science,Linguistics

What problem does this paper attempt to address?

VStyclone: Real-time Chinese voice style clone

Yichun Wu,Huihuang Zhao,Xiaoman Liang,Yaqi Sun

DOI: https://doi.org/10.1016/j.compeleceng.2022.108534

2022-12-22

Abstract:This paper proposes a novel Chinese speech cloning model named VStyclone, which consists of three stages: multi-speaker training, target speaker encoding, and target speaker synthesis. In this work, we design an efficient tone extractor, which can reallocate resources to the sequences of log-mel spectrogram frames obtained from multiple speakers' speech, thus allowing the network to learn multiple speakers' features differently. This approach allows the network to focus more on the voice features of the target speaker and extract the target features accurately. To cluster the voices of the same speaker and sparse the voices of different speakers, we build an optimal softmax loss to optimize the model. Then, we develop a style synthesizer, which adopts the idea of transformer instead of recurrent neural network, so that the model can not only process text information in parallel, but also improve the model's ability to process long-distance contextual information. Meanwhile, we embed a style extraction module in the style synthesizer to dynamically capture style ranges in an unsupervised manner. In addition, the VStyclone model uses generative adversarial networks as the base generation model of the vocoder to improve the generation speed, which runs 1.2 times faster than the real-time generation speed on CPU, and finally the VStyclone model obtains the SOTA effect.

engineering, electrical & electronic,computer science, interdisciplinary applications, hardware & architecture
A real-time voice cloning system with multiple algorithms for speech quality improvement

Weixin Hu,Xianyou Zhu

DOI: https://doi.org/10.1371/journal.pone.0283440

IF: 3.7

2023-04-04

PLoS ONE

Abstract:With the development of computer technology, speech synthesis techniques are becoming increasingly sophisticated. Speech cloning can be performed as a subtask of speech synthesis technology by using deep learning techniques to extract acoustic information from human voices and combine it with text to output a natural human voice. However, traditional speech cloning technology still has certain limitations; excessively large text inputs cannot be adequately processed, and the synthesized audio may include noise artifacts like breaks and unclear phrases. In this study, we add a text determination module to a synthesizer module to process words the model has not included. The original model uses fuzzy pronunciation for such words, which is not only meaningless but also affects the entire sentence. Thus, we improve the model by splitting the letters and pronouncing them separately. Finally, we also improved the preprocessing and waveform conversion modules of the synthesizer. We replace the pre-net module of the synthesizer and use an upgraded noise reduction algorithm combined with the SV2TTS framework to achieve a system with superior speech synthesis performance. Here, we focus on improving the performance of the synthesizer module to achieve higher-quality speech synthesis audio output.

multidisciplinary sciences
Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers

Zhaoyu Liu,Brian Mak

DOI: https://doi.org/10.48550/arXiv.1911.11601

2019-11-26

Audio and Speech Processing

Abstract:We investigate a novel cross-lingual multi-speaker text-to-speech synthesis approach for generating high-quality native or accented speech for native/foreign seen/unseen speakers in English and Mandarin. The system consists of three separately trained components: an x-vector speaker encoder, a Tacotron-based synthesizer and a WaveNet vocoder. It is conditioned on 3 kinds of embeddings: (1) speaker embedding so that the system can be trained with speech from many speakers will little data from each speaker; (2) language embedding with shared phoneme inputs; (3) stress and tone embedding which improves naturalness of synthesized speech, especially for a tonal language like Mandarin. By adjusting the various embeddings, MOS results show that our method can generate high-quality natural and intelligible native speech for native/foreign seen/unseen speakers. Intelligibility and naturalness of accented speech is low as expected. Speaker similarity is good for native speech from native speakers. Interestingly, speaker similarity is also good for accented speech from foreign speakers. We also find that normalizing speaker embedding x-vectors by L2-norm normalization or whitening improves output quality a lot in many cases, and the WaveNet performance seems to be language-independent: our WaveNet is trained with Cantonese speech and can be used to generate Mandarin and English speech very well.
DOP-Tacotron: a Fast Chinese TTS System with Local-based Attention

Ting He,Wei Zhao,Li Xu

DOI: https://doi.org/10.1109/ccdc49329.2020.9164203

2020-01-01

Abstract:As used in human-robot interaction, text-to-speech(TTS) systems can generate human-like speech from written text input to mimic human speakers. End-to-end TTS systems are widely explored in recent years. In this paper, we propose a fast trained end-to-end Chinese TTS system DOP-Tacotron. We propose the DOP module for the encoder and the post-processing network. DOP has almost similar effects with CBHG module while using 35.5% fewer parameters. We use local-based attention mechanism, which always follows the previous attention state. DOP-Tacotron achieves a 3.683 subjective 5-scale mean opinion score of naturalness on Chinese Mandarin, outperforming Tacotron in terms of naturalness. In addition, DOP-Tacotron adds stop-talk-loss to loss for spectrogram, and uses sample-length-batch for mini batch and accurate Chinese pinyin with punctuation as input. Our proposed TTS system can be easily trained since training time of DOP-Tacotron is only 2.5 hours.
Variational Auto-Encoder based Mandarin Speech Cloning

Qingyu Xing,Xiaohan Ma

DOI: https://doi.org/10.48550/arXiv.2203.02967

2022-03-06

Sound

Abstract:Speech cloning technology is becoming more sophisticated thanks to the advances in machine learning. Researchers have successfully implemented natural-sounding English speech synthesis and good English speech cloning by some effective models. However, because of prosodic phrasing and large character set of Mandarin, Chinese utilization of these models is not yet complete. By creating a new dataset and replacing Tacotron synthesizer with VAENAR-TTS, we improved the existing speech cloning technique CV2TTS to almost real-time speech cloning while guaranteeing synthesis quality. In the process, we customized the subjective tests of synthesis quality assessment by attaching various scenarios, so that subjects focus on the differences between voice and our improvements maybe were more advantageous to practical applications. The results of the A/B test, real-time factor (RTF) and 2.74 mean opinion score (MOS) in terms of naturalness and similarity, reflect the real-time high-quality Mandarin speech cloning we achieved.
OpenVoice: Versatile Instant Voice Cloning

Zengyi Qin,Wenliang Zhao,Xumin Yu,Xin Sun

2024-01-03

Abstract:We introduce OpenVoice, a versatile voice cloning approach that requires only a short audio clip from the reference speaker to replicate their voice and generate speech in multiple languages. OpenVoice represents a significant advancement in addressing the following open challenges in the field: 1) Flexible Voice Style Control. OpenVoice enables granular control over voice styles, including emotion, accent, rhythm, pauses, and intonation, in addition to replicating the tone color of the reference speaker. The voice styles are not directly copied from and constrained by the style of the reference speaker. Previous approaches lacked the ability to flexibly manipulate voice styles after cloning. 2) Zero-Shot Cross-Lingual Voice Cloning. OpenVoice achieves zero-shot cross-lingual voice cloning for languages not included in the massive-speaker training set. Unlike previous approaches, which typically require extensive massive-speaker multi-lingual (MSML) dataset for all languages, OpenVoice can clone voices into a new language without any massive-speaker training data for that language. OpenVoice is also computationally efficient, costing tens of times less than commercially available APIs that offer even inferior performance. To foster further research in the field, we have made the source code and trained model publicly accessible. We also provide qualitative results in our demo website. Prior to its public release, our internal version of OpenVoice was used tens of millions of times by users worldwide between May and October 2023, serving as the backend of MyShell.

Sound,Machine Learning,Audio and Speech Processing
U-Style: Cascading U-nets with Multi-level Speaker and Style Modeling for Zero-Shot Voice Cloning

Tao Li,Zhichao Wang,Xinfa Zhu,Jian Cong,Qiao Tian,Yuping Wang,Lei Xie

2023-10-06

Abstract:Zero-shot speaker cloning aims to synthesize speech for any target speaker unseen during TTS system building, given only a single speech reference of the speaker at hand. Although more practical in real applications, the current zero-shot methods still produce speech with undesirable naturalness and speaker similarity. Moreover, endowing the target speaker with arbitrary speaking styles in the zero-shot setup has not been considered. This is because the unique challenge of zero-shot speaker and style cloning is to learn the disentangled speaker and style representations from only short references representing an arbitrary speaker and an arbitrary style. To address this challenge, we propose U-Style, which employs Grad-TTS as the backbone, particularly cascading a speaker-specific encoder and a style-specific encoder between the text encoder and the diffusion decoder. Thus, leveraging signal perturbation, U-Style is explicitly decomposed into speaker- and style-specific modeling parts, achieving better speaker and style disentanglement. To improve unseen speaker and style modeling ability, these two encoders conduct multi-level speaker and style modeling by skip-connected U-nets, incorporating the representation extraction and information reconstruction process. Besides, to improve the naturalness of synthetic speech, we adopt mean-based instance normalization and style adaptive layer normalization in these encoders to perform representation extraction and condition adaptation, respectively. Experiments show that U-Style significantly surpasses the state-of-the-art methods in unseen speaker cloning regarding naturalness and speaker similarity. Notably, U-Style can transfer the style from an unseen source speaker to another unseen target speaker, achieving flexible combinations of desired speaker timbre and style in zero-shot voice cloning.

Sound,Audio and Speech Processing
Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning

Yu Zhang,Ron J. Weiss,Heiga Zen,Yonghui Wu,Zhifeng Chen,RJ Skerry-Ryan,Ye Jia,Andrew Rosenberg,Bhuvana Ramabhadran

DOI: https://doi.org/10.48550/arXiv.1907.04448

2019-07-24

Abstract:We present a multispeaker, multilingual text-to-speech (TTS) synthesis model based on Tacotron that is able to produce high quality speech in multiple languages. Moreover, the model is able to transfer voices across languages, e.g. synthesize fluent Spanish speech using an English speaker's voice, without training on any bilingual or parallel examples. Such transfer works across distantly related languages, e.g. English and Mandarin. Critical to achieving this result are: 1. using a phonemic input representation to encourage sharing of model capacity across languages, and 2. incorporating an adversarial loss term to encourage the model to disentangle its representation of speaker identity (which is perfectly correlated with language in the training data) from the speech content. Further scaling up the model by training on multiple speakers of each language, and incorporating an autoencoding input to help stabilize attention during training, results in a model which can be used to consistently synthesize intelligible speech for training speakers in all languages seen during training, and in native or foreign accents.

Computation and Language,Sound,Audio and Speech Processing
A Preliminary Study on Deep Learning-based Chinese Text to Taiwanese Speech Synthesis System

Y. Liao,Chen-Ming Pan,Wen-Han Hsu,Wern-Jun Wang,Cheng-Jung Tseng

Abstract:This paper focuses on the development and implementation of a Chinese Text-to-Taiwanese speech synthesis system. The proposed system combines three deep neural network-based modules including (1) a sequence-to-sequence-based Chinese characters to Taiwan Minnanyu Luomazi Pinyin (shortened to as Tâi-lô) machine translation (called C2T from now on), (2) a Tacotron2-based Tâi-lô pinyin to spectrogram and (3) a WaveGlow-based spectrogram to speech waveform synthesis subsystems. Among them, the C2T module was trained using a Chinese-Taiwanese parallel corpus (iCorpus) and 9 dictionaries released by Academia Sinica and collected from internet, respectively. The Tacotron2 and Waveglow was tuned using a Taiwanese speech synthesis corpus (a female speaker, about 10 hours speech) recorded by Chunghwa Telecom Laboratories. At the same time, a demonstration Chinese Text-to-Taiwanese speech synthesis web page has also been implemented. From the experimental results, it was found that (1) the best syllable error rate (SER) of 6.53% was achieved by the C2T module, (2) and the average MOS score of the whole speech synthesis system evaluated by 20 listeners gains 4.30. These results confirm that the effectiveness of integration of C2T, Tacrtron2 and WaveGlow models. In addition, the real-time factor of the whole system achieved 1/3.5.

Computer Science,Linguistics
Expressive Neural Voice Cloning

Paarth Neekhara,Shehzeen Hussain,Shlomo Dubnov,Farinaz Koushanfar,Julian McAuley

DOI: https://doi.org/10.48550/arXiv.2102.00151

2021-01-30

Abstract:Voice cloning is the task of learning to synthesize the voice of an unseen speaker from a few samples. While current voice cloning methods achieve promising results in Text-to-Speech (TTS) synthesis for a new voice, these approaches lack the ability to control the expressiveness of synthesized audio. In this work, we propose a controllable voice cloning method that allows fine-grained control over various style aspects of the synthesized speech for an unseen speaker. We achieve this by explicitly conditioning the speech synthesis model on a speaker encoding, pitch contour and latent style tokens during training. Through both quantitative and qualitative evaluations, we show that our framework can be used for various expressive voice cloning tasks using only a few transcribed or untranscribed speech samples for a new speaker. These cloning tasks include style transfer from a reference speech, synthesizing speech directly from text, and fine-grained style control by manipulating the style conditioning variables during inference.

Sound,Machine Learning,Audio and Speech Processing
Bidirectional Decoding Tacotron for Attention Based Neural Speech Synthesis

Wei Zhao,Li Xu

DOI: https://doi.org/10.1145/3561877.3561887

2022-01-01

Abstract:Attention-based neural text-to-speech (TTS) has become increasingly popular because of its end-to-end network architecture and impressive performance comparable to human recordings. However, existing approaches usually adopt a unidirectional decoding framework generating the target spectrum from left to right, which cannot take advantage of reverse target-side contexts from right to left. To mitigate the problem, we present a bidirectional decoding speech synthesis network based on the well-known Tacotron2. In particular, our model first employs a backward decoder to predict the spectrum from right to left, conditioned on the output states of the text encoder. Then, the forward decoder generates the spectrum from left to right, attending to both encoder outputs and the context hidden states from the backward decoder. With this architecture, our bidirectional decoding Tacotron2 can exploit both backward and forward information to promote the performance. Experiments with objective and subjective evaluations on LJSpeech have been conducted to demonstrate the effectiveness of our proposed method.
ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec

Shengpeng Ji,Jialong Zuo,Wen Wang,Minghui Fang,Siqi Zheng,Qian Chen,Ziyue Jiang,Hai Huang,Zehan Wang,Xize Cheng,Zhou Zhao

2024-10-23

Abstract:In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style, merely based on a few seconds of audio prompt and a simple textual style description prompt. Prior zero-shot TTS models and controllable TTS models either could only mimic the speaker's voice without further control and adjustment capabilities or were unrelated to speaker-specific voice generation. Therefore, ControlSpeech focuses on a more challenging new task-a TTS system with controllable timbre, content, and style at the same time. ControlSpeech takes speech prompts, content prompts, and style prompts as inputs and utilizes bidirectional attention and mask-based parallel decoding to capture corresponding codec representations in a discrete decoupling codec space. Moreover, we discovered the issue of text style controllability in a many-to-many mapping fashion and proposed the Style Mixture Semantic Density (SMSD) model to resolve this problem. SMSD module which is based on Gaussian mixture density networks, is designed to enhance the fine-grained partitioning and sampling capabilities of style semantic information and generate speech with more diverse styles. In terms of experiments, we make available a controllable model toolkit called ControlToolkit with a new style controllable dataset, some replicated baseline models and propose new metrics to evaluate both the control capability and the quality of generated audio in ControlSpeech. The relevant ablation studies validate the necessity of each component in ControlSpeech is necessary. We hope that ControlSpeech can establish the next foundation paradigm of controllable speech synthesis. The relevant code and demo are available at <a class="link-external link-https" href="https://github.com/jishengpeng/ControlSpeech" rel="external noopener nofollow">this https URL</a> .

Audio and Speech Processing,Machine Learning,Sound
A Chinese Text-to-Speech System

Weijun Chen,Jianmin Li,Fuzong Lin,Bo Zhang

DOI: https://doi.org/10.3321/j.issn:1002-8331.2000.09.001

2000-01-01

Abstract:This paper discusses the implementation of a typical Chinese Text-to-Speech op) system. Firstly, It presents the frame structure of the system and discusses all of the modules in detail, then analyzes its characteristics and problems existing. Lastly This paper does some experiments in order to improve the quality of the system: It puts forward a simple and effective algorithm to calculate the base period Of speech waveforms and verifis the importance of context in improving the naturalness of synthesis speech.
Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

Wei Fang,Yu-An Chung,James Glass

DOI: https://doi.org/10.48550/arxiv.1906.07307

2019-01-01

Abstract:Modern text-to-speech (TTS) systems are able to generate audio that sounds almost as natural as human speech. However, the bar of developing high-quality TTS systems remains high since a sizable set of studio-quality pairs is usually required. Compared to commercial data used to develop state-of-the-art systems, publicly available data are usually worse in terms of both quality and size. Audio generated by TTS systems trained on publicly available data tends to not only sound less natural, but also exhibits more background noise. In this work, we aim to lower TTS systems' reliance on high-quality data by providing them the textual knowledge extracted by deep pre-trained language models during training. In particular, we investigate the use of BERT to assist the training of Tacotron-2, a state of the art TTS consisting of an encoder and an attention-based decoder. BERT representations learned from large amounts of unlabeled text data are shown to contain very rich semantic and syntactic information about the input text, and have potential to be leveraged by a TTS system to compensate the lack of high-quality data. We incorporate BERT as a parallel branch to the Tacotron-2 encoder with its own attention head. For an input text, it is simultaneously passed into BERT and the Tacotron-2 encoder. The representations extracted by the two branches are concatenated and then fed to the decoder. As a preliminary study, although we have not found incorporating BERT into Tacotron-2 generates more natural or cleaner speech at a human-perceivable level, we observe improvements in other aspects such as the model is being significantly better at knowing when to stop decoding such that there is much less babbling at the end of the synthesized audio and faster convergence during training.
The Multi-Speaker Multi-Style Voice Cloning Challenge 2021

Qicong Xie,Xiaohai Tian,Guanghou Liu,Kun Song,Lei Xie,Zhiyong Wu,Hai Li,Song Shi,Haizhou Li,Fen Hong,Hui Bu,Xin Xu

DOI: https://doi.org/10.1109/icassp39728.2021.9414001

2021-01-01

Abstract:The Multi-speaker Multi-style Voice Cloning Challenge (M2VoC) aims to provide a common sizable dataset as well as a fair testbed for the benchmarking of the popular voice cloning task. Specifically, we formulate the challenge to adapt an average TTS model to the stylistic target voice with limited data from target speaker, evaluated by speaker identity and style similarity. The challenge consists of two tracks, namely few-shot track and one-shot track, where the participants are required to clone multiple target voices with 100 and 5 samples respectively. There are also two sub-tracks in each track. For sub-track a, to fairly compare different strategies, the participants are allowed to use only the training data provided by the organizer strictly. For sub-track b, the participants are allowed to use any data publicly available. In this paper, we present a detailed explanation on the tasks and data used in the challenge, followed by a summary of submitted systems and evaluation results.
Multi-speaker Chinese news broadcasting system based on improved Tacotron2

Wei Zhao,Yue Lian,Jianping Chai,Zhongwen Tu

DOI: https://doi.org/10.1007/s11042-023-15279-z

IF: 2.577

2023-05-04

Multimedia Tools and Applications

Abstract:In recent years, the demand for news broadcasting has increased with the explosion of information. The automatic news broadcasting system based on deep learning text-to-speech technology can solve the problems of working time limitation and errors caused by manual broadcasting. Most of the existing speech synthesis technologies cannot switch speakers in real-time and cannot solve a series of additional news broadcasting scenarios problems in Chinese. In this paper, we propose a multi-speaker Chinese news broadcasting system with switchable timbres based on our established Chinese news corpus CNews dataset for training. This system uses the CPM module to convert Chinese into pinyin phonemes more accurately. Then a timbre encoder is used to construct multi-speaker timbre feature embeddings. As for the problem of having long texts in news, the acoustic model of this system is improved based on Tacotron2 and uses Discrete Grave attention as the attention mechanism so that the model reduces the demand for audio data in the training phase and better extracts the information from the context. The HiFi-GAN vocoder is also used to generate time domain waveforms instead of the original WaveNet, reducing the synthesis time and improving the voice quality of the synthesized speech. Experiments show that the system can change the target timbre flexibly according to the reference speech compared with Tacotron2. Moreover, it is able to synthesize speech with the prosody and style of the target presenter under the training of limited data and with better naturalness as well as faster inference speed, which can be used for real-time news broadcasting.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Teacher-Student Training For Robust Tacotron-Based TTS

Rui Liu,Berrak Sisman,Jingdong Li,Feilong Bao,Guanglai Gao,Haizhou Li

DOI: https://doi.org/10.1109/icassp40776.2020.9054681

2020-05-01

Abstract:While neural end-to-end text-to-speech (TTS) is superior to conventional statistical methods in many ways, the exposure bias problem in the autoregressive models remains an issue to be resolved. The exposure bias problem arises from the mismatch between the training and inference process, that results in unpredictable performance for out-of-domain test data at run-time. To overcome this, we propose a teacher-student training scheme for Tacotron-based TTS by introducing a distillation loss function in addition to the feature loss function. We first train a Tacotron2-based TTS model by always providing natural speech frames to the decoder, that serves as a teacher model. We then train another Tacotron2-based model as a student model, of which the decoder takes the predicted speech frames as input, similar to how the decoder works during run-time inference. With the distillation loss, the student model learns the output probabilities from the teacher model, that is called knowledge distillation. Experiments show that our proposed training scheme consistently improves the voice quality for out-of-domain test data both in Chinese and English systems.
Multi-modal Adversarial Training for Zero-Shot Voice Cloning

John Janiczek,Dading Chong,Dongyang Dai,Arlo Faria,Chao Wang,Tao Wang,Yuzong Liu

2024-08-29

Abstract:A text-to-speech (TTS) model trained to reconstruct speech given text tends towards predictions that are close to the average characteristics of a dataset, failing to model the variations that make human speech sound natural. This problem is magnified for zero-shot voice cloning, a task that requires training data with high variance in speaking styles. We build off of recent works which have used Generative Advsarial Networks (GAN) by proposing a Transformer encoder-decoder architecture to conditionally discriminates between real and generated speech features. The discriminator is used in a training pipeline that improves both the acoustic and prosodic features of a TTS model. We introduce our novel adversarial training technique by applying it to a FastSpeech2 acoustic model and training on Libriheavy, a large multi-speaker dataset, for the task of zero-shot voice cloning. Our model achieves improvements over the baseline in terms of speech quality and speaker similarity. Audio examples from our system are available online.

Audio and Speech Processing,Machine Learning,Sound
End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Yuewen Cao,Xixin Wu,Songxiang Liu,Jianwei Yu,Xu Li,Zhiyong Wu,Xunying Liu,Helen Meng

DOI: https://doi.org/10.1109/icassp.2019.8682927

2019-01-01

Abstract:State-of-the-art text-to-speech (TTS) synthesis models can produce monolingual speech with high intelligibility and naturalness. However, when the models are applied to synthesize code-switched (CS) speech, the performance declines seriously. Conventionally, developing a CS TTS system requires multilingual data to incorporate language-specific and cross-lingual knowledge. Recently, end-to-end (E2E) architecture has achieved satisfactory results in monolingual TTS. The architecture enables the training from one end of alphabetic text input to the other end of acoustic feature output. In this paper, we explore the use of E2E framework for CS TTS, using a combination of Mandarin and English monolingual speech corpus uttered by two female speakers. To handle alphabetic input from different languages, we explore two kinds of encoders: (1) shared multilingual encoder with explicit language embedding (LDE); (2) separated monolingual encoder (SPE) for each language. The two systems use identical decoder architecture, where a discriminative code is incorporated to enable the model to generate speech in one speaker's voice consistently. Experiments confirm the effectiveness of the proposed modifications on the E2E TTS framework in terms of quality and speaker similarity of the generated speech. Moreover, our proposed systems can generate controllable foreign-accented speech at character-level using only mixture of monolingual training data.
UNET-TTS: Improving Unseen Speaker and Style Transfer in One-Shot Voice Cloning

Rui Li,Dong Pu,Minnie Huang,Bill Huang

DOI: https://doi.org/10.1109/icassp43922.2022.9746049

2022-05-23

Abstract:One-shot voice cloning aims to transform speaker voice and speaking style in speech synthesized from a text-to-speech (TTS) system, where only a shot recording from the target reference speech can be used. Out-of-domain transfer is still a challenging task, and one important aspect that impacts the accuracy and similarity of synthetic speech is the conditional representations carrying speaker or style cues extracted from the limited references. In this paper, we present a novel one-shot voice cloning algorithm called Unet-TTS that has good generalization ability for unseen speakers and styles. Based on a skip-connected U-net structure, the new model can efficiently discover speaker-level and utterance-level spectral feature details from the reference audio, enabling accurate inference of complex acoustic characteristics as well as imitation of speaking styles into the synthetic speech. According to both subjective and objective evaluations of similarity, the new model outperforms both speaker embedding and unsupervised style modeling (GST) approaches on an unseen emotional corpus.

Voice Style Cloning for Chinese Speech

VStyclone: Real-time Chinese voice style clone

A real-time voice cloning system with multiple algorithms for speech quality improvement

Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers

DOP-Tacotron: a Fast Chinese TTS System with Local-based Attention

Variational Auto-Encoder based Mandarin Speech Cloning

OpenVoice: Versatile Instant Voice Cloning

U-Style: Cascading U-nets with Multi-level Speaker and Style Modeling for Zero-Shot Voice Cloning

Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning

A Preliminary Study on Deep Learning-based Chinese Text to Taiwanese Speech Synthesis System

Expressive Neural Voice Cloning

Bidirectional Decoding Tacotron for Attention Based Neural Speech Synthesis

ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec

A Chinese Text-to-Speech System

Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

The Multi-Speaker Multi-Style Voice Cloning Challenge 2021

Multi-speaker Chinese news broadcasting system based on improved Tacotron2

Teacher-Student Training For Robust Tacotron-Based TTS

Multi-modal Adversarial Training for Zero-Shot Voice Cloning

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

UNET-TTS: Improving Unseen Speaker and Style Transfer in One-Shot Voice Cloning