Abstract:This paper presents a new spectral envelope conversion method using deep neural networks (DNNs). The conventional joint density Gaussian mixture model (JDGMM) based spectral conversion methods perform stably and effectively. However, the speech generated by these methods suffer severe quality degradation due to the following two factors: 1) inadequacy of JDGMM in modeling the distribution of spectral features as well as the non-linear mapping relationship between the source and target speakers, 2) spectral detail loss caused by the use of high-level spectral features such as mel-cepstra. Previously, we have proposed to use the mixture of restricted Boltzmann machines (MoRBM) and the mixture of Gaussian bidirectional associative memories (MoGBAM) to cope with these problems. In this paper, we propose to use a DNN to construct a global non-linear mapping relationship between the spectral envelopes of two speakers. The proposed DNN is generatively trained by cascading two RBMs, which model the distributions of spectral envelopes of source and target speakers respectively, using a Bernoulli BAM (BBAM). Therefore, the proposed training method takes the advantage of the strong modeling ability of RBMs in modeling the distribution of spectral envelopes and the superiority of BAMs in deriving the conditional distributions for conversion. Careful comparisons and analysis among the proposed method and some conventional methods are presented in this paper. The subjective results show that the proposed method can significantly improve the performance in terms of both similarity and naturalness compared to conventional methods.

The USTC System for Voice Conversion Challenge 2016: Neural Network Based Approaches for Spectrum, Aperiodicity and F0 Conversion

VITS-based Singing Voice Conversion System with DSPGAN post-processing for SVCC2023

Sequence-to-Sequence Acoustic Modeling for Voice Conversion

Non-Parallel Voice Conversion with Autoregressive Conversion Model and Duration Adjustment

Voice Conversion Using Deep Neural Networks with Layer-Wise Generative Training

The NUS & NWPU System for Voice Conversion Challenge 2020

CASIA Voice Conversion System for the Voice Conversion Challenge 2020

High-Quality Voice Conversion Using Spectrogram-Based Wavenet Vocoder

Voice Conversion Using Deep Neural Network in Super-Frame Feature Space

Voice Conversion Using Generative Trained Deep Neural Networks with Multiple Frame Spectral Envelopes

The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods.

Articulatory-to-Acoustic Conversion with Cascaded Prediction of Spectral and Excitation Features Using Neural Networks

Spectro-Temporal Modelling with Time-Frequency Lstm and Structured Output Layer for Voice Conversion

The Academia Sinica Systems of Voice Conversion for VCC2020

Phone-aware LSTM-RNN for Voice Conversion

U2-VC: One-Shot Voice Conversion Using Two-Level Nested U-structure

Voice Conversion Challenge 2020: Intra-lingual Semi-Parallel and Cross-Lingual Voice Conversion

A Compact Framework For Voice Conversion Using Wavenet Conditioned On Phonetic Posteriorgrams

Pitch Transformation in Neural Network Based Voice Conversion

Voice Conversion with High Naturalness Using Spectrum and Super-segmental Feature Transform

Multimodal Voice Conversion under Adverse Environment Using a Deep Convolutional Neural Network