Abstract:Amethod of learning andmodeling unit embeddings using deep neutral networks (DNNs) is presented in this article for unit-selection-based Mandarin speech synthesis. Here, a unit embedding is defined as a fixed-length embedding vector for a phone-sized unit candidate in a corpus. Modeling phone-sized embedding vectors instead of frame-sized acoustic features can better measure the long-term dependencies among consecutive units in an utterance. First, a DNN with an embedding layer is built to learn the embedding vectors of all unit candidates in the corpus from scratch. In order to enable the extracted embedding vectors to carry both acoustic and linguistic information of unit candidates, a multitarget learning strategy is designed for the DNN. Its optional prediction targets include frame-level acoustic features, unit durations, monophone and tone identifiers, and context classes. Then, another two DNNs are constructed to map linguistic features toward the extracted embedding vectors. One of them employs the unit vectors of preceding phones besides the linguistic features of current phone as its input. At synthesis time, the distances between the unit vectors predicted by these two DNNs and the ones derived from unit candidates are used as a part of the target cost and a part of the concatenation cost, respectively. Our experiments on a Mandarin speech synthesis corpus demonstrate that learning and modeling unit embeddings improve the naturalness of hidden Markov model (HMM)-based unit selection speech synthesis. Furthermore, integrating multiple targets for learning unit embeddings achieves better performance than using only acoustic targets according to our subjective evaluation results.

Progressive Neural Networks Based Features Prediction for the Target Cost in Unit-Selection Speech Synthesizer

Transfer Learning Based Progressive Neural Networks for Acoustic Modeling in Statistical Parametric Speech Synthesis.

Deep Metric Learning For The Target Cost In Unit-Selection Speech Synthesizer

DNN-based unit selection using frame-sized speech segments

Unit Selection Speech Synthesis Using Frame-Sized Speech Segments and Neural Network Based Acoustic Models

Extracting Unit Embeddings Using Sequence-To-Sequence Acoustic Models for Unit Selection Speech Synthesis

Context features based pre-selection and weight prediction in concatenation speech synthesis system

Deep Neural Network Based Noised Asian Speech Enhancement and Its Implementation on a Hearing Aid App.

Learning and Modeling Unit Embeddings Using Deep Neural Networks for Unit-Selection-Based Mandarin Speech Synthesis.

SNR-Based Progressive Learning of Deep Neural Network for Speech Enhancement

Improving Deep Neural Network Based Speech Synthesis Through Contextual Feature Parametrization and Multi-Task Learning

A data driven method for target and concatenation cost calculation with KL-Divergence in Mandarin hybrid speech synthesis

Articulatory-to-Acoustic Conversion with Cascaded Prediction of Spectral and Excitation Features Using Neural Networks

Learning and Modeling Unit Embeddings for Improving HMM-based Unit Selection Speech Synthesis

Deep Time Delay Neural Network for Speech Enhancement with Full Data Learning

Improved unit selection speech synthesis method utilizing subjective evaluation results on synthetic speech

Progressive Multi-Target Network Based Speech Enhancement with Snr-Preselection for Robust Speaker Diarization

Patnet : A Phoneme-Level Autoregressive Transformer Network for Speech Synthesis.

Perceptual Clustering Based Unit Selection Optimization for Concatenative Text-to-speech Synthesis

Stage-Wise and Prior-Aware Neural Speech Phase Prediction

Multi-Stage Progressive Speech Enhancement Network