Abstract:This paper presents a new spectral envelope conversion method using deep neural networks (DNNs). The conventional joint density Gaussian mixture model (JDGMM) based spectral conversion methods perform stably and effectively. However, the speech generated by these methods suffer severe quality degradation due to the following two factors: 1) inadequacy of JDGMM in modeling the distribution of spectral features as well as the non-linear mapping relationship between the source and target speakers, 2) spectral detail loss caused by the use of high-level spectral features such as mel-cepstra. Previously, we have proposed to use the mixture of restricted Boltzmann machines (MoRBM) and the mixture of Gaussian bidirectional associative memories (MoGBAM) to cope with these problems. In this paper, we propose to use a DNN to construct a global non-linear mapping relationship between the spectral envelopes of two speakers. The proposed DNN is generatively trained by cascading two RBMs, which model the distributions of spectral envelopes of source and target speakers respectively, using a Bernoulli BAM (BBAM). Therefore, the proposed training method takes the advantage of the strong modeling ability of RBMs in modeling the distribution of spectral envelopes and the superiority of BAMs in deriving the conditional distributions for conversion. Careful comparisons and analysis among the proposed method and some conventional methods are presented in this paper. The subjective results show that the proposed method can significantly improve the performance in terms of both similarity and naturalness compared to conventional methods.

Recurrent neural network for spectral mapping in speech bandwidth extension

Speech Bandwidth Extension Using Recurrent Temporal Restricted Boltzmann Machines

Speech Bandwidth Extension Using Bottleneck Features and Deep Recurrent Neural Networks.

Restoring High Frequency Spectral Envelopes Using Neural Networks For Speech Bandwidth Extension

Speech Bandwidth Extension Based on Restricted Boltzmann Machines

Using conditional restricted Boltzmann machines for spectral envelope modeling in speech bandwidth extension.

Waveform Modeling and Generation Using Hierarchical Recurrent Neural Networks for Speech Bandwidth Extension

Speech bandwidth extension supported by temporal information

A Novel Unified Framework for Speech Enhancement and Bandwidth Extension Based on Jointly Trained Neural Networks

Audio Bandwidth Extension Using Ensemble of Recurrent Neural Networks

Speech Bandwidth Expansion Based on Deep Neural Networks

A Novel Research to Artificial Bandwidth Extension Based on Deep BLSTM Recurrent Neural Networks and Exemplar-Based Sparse Representation.

Spectral Envelope Estimation Used for Audio Bandwidth Extension Based on RBF Neural Network

Audio bandwidth extension based on RBF neural network

High Fidelity Speech Enhancement with Band-split RNN

An RNN-based Speech Enhancement Method for a Binaural Hearing Aid System

Real-Time Speech Enhancement Algorithm of Recurrent Neural Network Based on Energy Spectrum Depth Modulation

A Dual-Branch Neural Network for Phase-Aware Speech Bandwidth Extension

DSP-informed bandwidth extension using locally-conditioned excitation and linear time-varying filter subnetworks

Speech Bandwidth Extension Based on Codebook Mapping and GMM

Voice Conversion Using Deep Neural Networks with Layer-Wise Generative Training