Abstract:This paper proposes an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel voice conversion approach, which utilizes text supervision during training. In this approach, we combine a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. During the training stage, an encoder-decoder-based hybrid connectionist-temporal-classification-attention (CTC-attention) phoneme recognizer is trained, whose encoder has a bottle-neck layer. A BNE is obtained from the phoneme recognizer and is utilized to extract speaker-independent, dense and rich spoken content representations from spectral features. Then a multi-speaker location-relative attention based seq2seq synthesis model is trained to reconstruct spectral features from the bottle-neck features, conditioning on speaker representations for speaker identity control in the generated speech. To mitigate the difficulties of using seq2seq models to align long sequences, we down-sample the input spectral feature along the temporal dimension and equip the synthesis model with a discretized mixture of logistic (MoL) attention mechanism. Since the phoneme recognizer is trained with large speech recognition data corpus, the proposed approach can conduct any-to-many voice conversion. Objective and subjective evaluations show that the proposed any-to-many approach has superior voice conversion performance in terms of both naturalness and speaker similarity. Ablation studies are conducted to confirm the effectiveness of feature selection and model design strategies in the proposed approach. The proposed VC approach can readily be extended to support any-to-any VC (also known as one/few-shot VC), and achieve high performance according to objective and subjective evaluations.

Building Multi lingual TTS using Cross Lingual Voice Conversion

Building Bilingual and Code-Switched Voice Conversion with Limited Training Data Using Embedding Consistency Loss

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Cross-lingual Voice Conversion with Disentangled Universal Linguistic Representations

Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling

Voice Conversion Challenge 2020: Intra-lingual Semi-Parallel and Cross-Lingual Voice Conversion

A Modularized Neural Network with Language-Specific Output Layers for Cross-lingual Voice Conversion

Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech

ViSPer: A Multilingual TTS Approach Based on VITS Using Deep Feature Loss

Towards General-Purpose Text-Instruction-Guided Voice Conversion

Transfer Learning from Speech Synthesis to Voice Conversion with Non-Parallel Training Data

Non-Parallel Voice Conversion with Autoregressive Conversion Model and Duration Adjustment

Non-parallel Sequence-to-Sequence Voice Conversion for Arbitrary Speakers.

VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion

Enhancing Polyglot Voices by Leveraging Cross-Lingual Fine-Tuning in Any-to-One Voice Conversion

Voice Conversion by Cascading Automatic Speech Recognition and Text-to-Speech Synthesis with Prosody Transfer.

Multi-Target Emotional Voice Conversion With Neural Vocoders

Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion

Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation

Disentangling Content and Fine-Grained Prosody Information Via Hybrid ASR Bottleneck Features for Voice Conversion

The Academia Sinica Systems of Voice Conversion for VCC2020