Learning and Modeling Unit Embeddings Using Deep Neural Networks for Unit-Selection-Based Mandarin Speech Synthesis

ZhouXiao,LingZhen-Hua,DaiLi-Rong

2020-01-01

Abstract:A method of learning and modeling unit embeddings using deep neutral networks (DNNs) is presented in this article for unit-selection-based Mandarin speech synthesis. Here, a unit embedding is defin...

What problem does this paper attempt to address?

Learning and Modeling Unit Embeddings Using Deep Neural Networks for Unit-Selection-Based Mandarin Speech Synthesis.

Xiao Zhou,Zhen-Hua Ling,Li-Rong Dai

DOI: https://doi.org/10.1145/3372244

IF: 1.471

2020-01-01

ACM Transactions on Asian and Low-Resource Language Information Processing

Abstract:Amethod of learning andmodeling unit embeddings using deep neutral networks (DNNs) is presented in this article for unit-selection-based Mandarin speech synthesis. Here, a unit embedding is defined as a fixed-length embedding vector for a phone-sized unit candidate in a corpus. Modeling phone-sized embedding vectors instead of frame-sized acoustic features can better measure the long-term dependencies among consecutive units in an utterance. First, a DNN with an embedding layer is built to learn the embedding vectors of all unit candidates in the corpus from scratch. In order to enable the extracted embedding vectors to carry both acoustic and linguistic information of unit candidates, a multitarget learning strategy is designed for the DNN. Its optional prediction targets include frame-level acoustic features, unit durations, monophone and tone identifiers, and context classes. Then, another two DNNs are constructed to map linguistic features toward the extracted embedding vectors. One of them employs the unit vectors of preceding phones besides the linguistic features of current phone as its input. At synthesis time, the distances between the unit vectors predicted by these two DNNs and the ones derived from unit candidates are used as a part of the target cost and a part of the concatenation cost, respectively. Our experiments on a Mandarin speech synthesis corpus demonstrate that learning and modeling unit embeddings improve the naturalness of hidden Markov model (HMM)-based unit selection speech synthesis. Furthermore, integrating multiple targets for learning unit embeddings achieves better performance than using only acoustic targets according to our subjective evaluation results.
Learning and Modeling Unit Embeddings for Improving HMM-based Unit Selection Speech Synthesis

Xiao Zhou,Zhen-Hua Ling,Zhi-Ping Zhou,Li-Rong Dai

DOI: https://doi.org/10.21437/interspeech.2018-1198

2018-01-01

Abstract:This paper presents a method of learning and modeling unit embeddings using deep neutral networks (DNNs) to improve the performance of HMM-based unit selection speech synthesis. First, a DNN with an embedding layer is built to learn a fixed-length embedding vector for each phone-sized candidate unit in the corpus from scratch. Then, another two DNNs are constructed to map linguistic features toward the extracted unit vector of each phone. One of them employs the unit vectors of preceding phones as model input. At synthesis time, the L-2 distances between the unit vectors predicted by these two DNNs and the ones derived from candidate units are integrated into the target cost and the concatenation cost of HMM-based unit selection speech synthesis respectively. Experimental results demonstrate that the unit vectors estimated using only acoustic features display phone-dependent clustering properties. Furthermore, integrating unit vector distances into cost functions, especially the concatenation cost, improves the naturalness of HMM-based unit selection speech synthesis in our experiments.
Extracting Unit Embeddings Using Sequence-To-Sequence Acoustic Models for Unit Selection Speech Synthesis

Xiao Zhou,Zhen-Hua Ling,Li-Rong Dai

DOI: https://doi.org/10.1109/icassp40776.2020.9053812

2020-01-01

Abstract:This paper presents a method of using the intermediate representations between linguistic and acoustic features in a Tacotron model to derive the cost functions for unit selection speech synthesis. By extracting the outputs of the Tacotron encoder, each phone-sized candidate unit in the corpus is represented by a fixed-length unit vector. Similarly, each target unit to be synthesized is also converted into a unit vector of the same dimension by encoding the input phone sequence. The normalized Euclidean distances between these two vectors are utilized to fulfill unit pre-selection and to calculate the target cost for unit selection. Then, another DNN which predicts the unit vector of each phone from its preceding ones is constructed to derive the concatenation cost function. Experimental results demonstrate that the unit vectors extracted from Tacotron contain both duration and acoustic information of phone units. Comparing with our previous work, which learned unit vectors using a DNN and only acoustic features, the method proposed in this paper further improves the naturalness of unit selection speech synthesis in our experiments.
DNN-based unit selection using frame-sized speech segments

Zhi-Ping Zhou,Zhen-Hua Ling

DOI: https://doi.org/10.1109/ISCSLP.2016.7918404

2016-01-01

Abstract:This paper presents a deep neural network (DNN)-based unit selection method for waveform concatenation speech synthesis using frame-sized speech segments. In this method, three DNNs are adopted to calculate target costs and concatenation costs respectively for selecting frame-sized candidate units. The first DNN is built in the same way as the DNN-based statistical parametric speech synthesis, which predicts target acoustic features given linguistic context inputs. The distance between the acoustic features of a candidate unit and the predicted ones for a target unit is calculated as the target cost. The other two DNNs are constructed to predict the acoustic features at current frame using its context features and the acoustic features of previous frames. At synthesis time, these two DNNs are employed to calculate the concatenation cost for each candidate unit given its preceding units. Experimental results show that our proposed method can achieve better naturalness than the hidden Markov model (HMM)-based frame selection method and the HMM-based parametric synthesis method.
Unit Selection Speech Synthesis Using Frame-Sized Speech Segments and Neural Network Based Acoustic Models

Zhen-Hua Ling,Zhi-Ping Zhou

DOI: https://doi.org/10.1007/s11265-018-1336-0

2018-01-01

Abstract:This paper proposes to select frame-sized speech segments for waveform concatenation speech synthesis using neural network based acoustic models. First, a deep neural network (DNN) based frame selection method is presented. In this method, three DNNs are adopted to calculate target costs and concatenation costs respectively for selecting candidate frames of 5ms length. One DNN is built in the same way as the DNN-based statistical parametric speech synthesis, which predicts target acoustic features given linguistic context inputs. The distance between the acoustic features of a candidate unit and the predicted ones for a target unit is calculated as the target cost. The other two DNNs are constructed to predict the acoustic features at current frame using its context features and the acoustic features of preceding frames. At synthesis time, these two DNNs are employed to calculate the concatenation cost for each candidate frame given its preceding ones. Furthermore, recurrent neural networks (RNNs) with long short-term memory (LSTM) cells are adopted to replace DNNs for acoustic modeling in order to make better use of the sequential information. A strategy of using multi-frame instead of single frame as the basic unit for selection is also presented to reduce the concatenation points within synthetic speech. Experimental results show that our proposed method can achieve better naturalness than the hidden Markov model (HMM)-based frame selection method and the HMM-based parametric speech synthesis method.
Unitnet: A Sequence-To-Sequence Acoustic Model For Concatenative Speech Synthesis

Xiao Zhou,Zhen-Hua Ling,Li-Rong Dai

DOI: https://doi.org/10.1109/TASLP.2021.3093823

2021-01-01

Abstract:This paper presents UnitNet, a sequence-to-sequence (Seq2Seq) acoustic model for concatenative speech synthesis. Comparing with the Tacotron2 model for Seq2Seq speech synthesis, UnitNet utilizes the phone boundaries of training data and its decoder contains autoregressive structures at both phone and frame levels. This hierarchical architecture can not only extract embedding vectors for representing phone-sized units in the corpus but also measure the dependency among consecutive units, which makes the UnitNet model capable of guiding the selection of phone-sized units for concatenative speech synthesis. A byproduct of this model is that it can also be applied to statistical parametric speech synthesis (SPSS) and improve the robustness of Seq2Seq acoustic feature prediction since it adopts interpretable transition probability prediction rather than attention mechanism for frame-level alignment. Experimental results show that our UnitNet-based concatenative speech synthesis method not only outperforms the unit selection methods using hidden Markov models and Tacotron-based unit embeddings, but also achieves better naturalness and faster inference speed than the SPSS method using FastSpeech and Parallel WaveGAN. Besides, the UnitNet-based SPSS method makes fewer synthesis errors than Tacotron2 and FastSpeech without naturalness degradation.
Selection of acoustic modeling unit for Tibetan speech recognition based on deep learning

Baojia Gong,Rangzhuoma Cai,Zhijie Cai,Yuntao Ding,Maozhaxi Peng

DOI: https://doi.org/10.1051/matecconf/202133606014

2021-01-01

MATEC Web of Conferences

Abstract:The selection of the speech recognition modeling unit is the primary problem of acoustic modeling in speech recognition, and different acoustic modeling units will directly affect the overall performance of speech recognition. This paper designs the Tibetan character segmentation and labeling model and algorithm flow for the purpose of solving the problem of selecting the acoustic modeling unit in Tibetan speech recognition by studying and analyzing the deficiencies of the existing acoustic modeling units in Tibetan speech recognition. After experimental verification, the Tibetan character segmentation and labeling model and algorithm achieved good performance of character segmentation and labeling, and the accuracy of Tibetan character segmentation and labeling reached 99.98%, respectively.

English Else
Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition

Yuting Yang,Binbin Du,Yuke Li

DOI: https://doi.org/10.48550/arXiv.2205.11998

2022-10-18

Abstract:The choice of modeling units is crucial for automatic speech recognition (ASR) tasks. In mandarin scenarios, the Chinese characters represent meaning but are not directly related to the pronunciation. Thus only considering the writing of Chinese characters as modeling units is insufficient to capture speech features. In this paper, we present a novel method involves with multi-level modeling units, which integrates multi-level information for mandarin speech recognition. Specifically, the encoder block considers syllables as modeling units and the decoder block deals with character-level modeling units. To facilitate the incremental conversion from syllable features to character features, we design an auxiliary task that applies cross-entropy (CE) loss to intermediate decoder layers. During inference, the input feature sequences are converted into syllable sequences by the encoder block and then converted into Chinese characters by the decoder block. Experiments on the widely used AISHELL-1 corpus demonstrate that our method achieves promising results with CER of 4.1%/4.6% and 4.6%/5.2%, using the Conformer and the Transformer backbones respectively.

Computation and Language,Sound,Audio and Speech Processing
A comparative study on selecting acoustic modeling units in deep neural networks based large vocabulary Chinese speech recognition

Xiangang Li,Yuning Yang,Zaihu Pang,Xihong Wu

DOI: https://doi.org/10.1016/j.neucom.2014.07.087

IF: 6

2015-12-01

Neurocomputing

Abstract:This paper compared the performance of different acoustic modeling units in deep neural networks (DNNs) based large vocabulary continuous speech recognition (LVCSR) systems for Chinese. Recently, the deep neural networks based acoustic modeling method has achieved very competitive performance for many speech recognition tasks, and has become the focus of current LVCSR research. Some previous work have studied the context independent and context dependent DNNs based acoustic models. For Chinese, a syllabic language, the choice of basic modeling units under the background of DNNs based LVCSR systems is a very important issue.Three basic modeling units, syllables, initial/finals, phones, are discussed and compared. Experimental results show that, in the DNNs based systems, the context dependent (CD) phones obtain the best performance, and the context independent (CI) syllables have the similar performance with the CD initial/finals. How the number of clustered states impacts on the performance of DNNs based systems is also discussed, which showed different properties from the GMMs based systems. Besides, through introducing the multi-task learning strategy, these multiple modeling units can be combined in the DNNs training procedure. The experimental results indicate that combining these multiple modeling units using multi-task learning outperforms each individual modeling unit.

computer science, artificial intelligence
Statistical Acoustic Model Based Unit Selection Algorithm for Speech Synthesis

Zhen-Hua LING,Ren-Hua WANG

DOI: https://doi.org/10.3969/j.issn.1003-6059.2008.03.002

2008-01-01

Abstract:A statistical acoustic model based unit selection algorithm for speech synthesis is proposed. During training stage, the acoustic models for contextual dependent phonemes are built up by using acoustic features extracted from the training data, such as spectral parameters, F0, and segmental and prosodic labels in the corpus. The hidden Markov model (HMM) is adopted as the model structure. During synthesis stage, the optimal phoneme unit sequence is searched in the speech corpus by maximizing the probabilistic likelihood between its acoustic features and the sentence HMM constructed with the contextual information of input text. Finally, the waveforms of the selected candidate units are concatenated and smoothed to produce the synthesized speech. Based on the proposed method, a Chinese speech synthesis system using initials and finals as the basic concatenation units is constructed. Results of listening test prove that the proposed method can achieve better naturalness of synthesized speech compared to the conventional method.
Deep Metric Learning For The Target Cost In Unit-Selection Speech Synthesizer

Ruibo Fu,Jianhua Tao,Yibin Zheng,Zhengqi Wen

DOI: https://doi.org/10.21437/Interspeech.2018-1305

2018-01-01

Abstract:This paper describes a unified Deep Metric Learning (DML) framework to predict the target cost directly by supervised learning method. The conventional methods to calculate the target cost include two separate steps: feature extraction and standard distance measurement. The proposed DML framework aims to measure the similarity between the candidate units and the target units more reasonably and directly. Firstly, the symmetrical DML framework is pre-trained to learn the metric between pairs of candidate units and target units. The relabeling procedure is added to correct the initial designed labels of the target cost. Secondly, the acoustic features of the target units are removed, which fits the runtime of the unit-selection synthesizer. The asymmetrical DML is fine-tuned to learn the metric between candidate units and target units. Compared with the conventional methods, the proposed unified DML framework can avoid the accumulation of errors in separate steps and improve the accuracy in labeling and predicting the target cost. The evaluation results demonstrate that the naturalness of synthetic speech has been improved by adopting DML framework to predict target cost.
Trainable Unit Selection Speech Synthesis under Statistical Framework

RenHua Wang,LiRong Dai,ZhenHua Ling,Yu Hu

DOI: https://doi.org/10.1007/s11434-009-0267-3

2009-01-01

Chinese Science Bulletin (Chinese Version)

Abstract:This paper proposes a trainable unit selection speech synthesis method based on statistical modeling framework. At training stage, acoustic features are extracted from the training database and statistical models are estimated for each feature. During synthesis, the optimal candidate unit sequence is searched out from the database following the maximum likelihood criterion derived from the trained models. Finally, the waveforms of the optimal candidate units are concatenated to produce synthetic speech. Experiment results show that this method can improve the automation of system construction and naturalness of synthetic speech effectively compared with the conventional unit selection synthesis method. Furthermore, this paper presents a minimum unit selection error model training criterion according to the characteristics of unit selection speech synthesis and adopts discriminative training for model parameter estimation. This criterion can finally achieve the full automation of system construction and improve the naturalness of synthetic speech further.
State-Clustering Based Multiple Deep Neural Networks Modeling Approach for Speech Recognition

Pan Zhou,Hui Jiang,Li-Rong Dai,Yu Hu,Qingfeng Liu

DOI: https://doi.org/10.1109/TASLP.2015.2392944

2015-01-01

Abstract:The hybrid deep neural network (DNN) and hidden Markov model (HMM) has recently achieved dramatic performance gains in automatic speech recognition (ASR). The DNN-based acoustic model is very powerful but its learning process is extremely time-consuming. In this paper, we propose a novel DNN-based acoustic modeling framework for speech recognition, where the posterior probabilities of HMM states are computed from multiple DNNs (mDNN), instead of a single large DNN, for the purpose of parallel training towards faster turnaround. In the proposed mDNN method all tied HMM states are first grouped into several disjoint clusters based on data-driven methods. Next, several hierarchically structured DNNs are trained separately in parallel for these clusters using multiple computing units (e.g. GPUs). In decoding, the posterior probabilities of HMM states can be calculated by combining outputs from multiple DNNs. In this work, we have shown that the training procedure of the mDNN under popular criteria, including both frame-level cross-entropy and sequence-level discriminative training, can be parallelized efficiently to yield significant speedup. The training speedup is mainly attributed to the fact that multiple DNNs are parallelized over multiple GPUs and each DNN is smaller in size and trained by only a subset of training data. We have evaluated the proposed mDNN method on a 64-hour Mandarin transcription task and the 320-hour Switchboard task. Compared to the conventional DNN, a 4-cluster mDNN model with similar size can yield comparable recognition performance in Switchboard (only about 2% performance degradation) with a greater than 7 times speed improvement in CE training and a 2.9 times improvement in sequence training, when 4 GPUs are used.
A regression approach to speech enhancement based on deep neural networks

Yong Xu,Jun Du,Li-Rong Dai,Chin-Hui Lee

DOI: https://doi.org/10.1109/TASLP.2014.2364452

2015-01-01

Abstract:In contrast to the conventional minimum mean square error (MMSE)-based noise reduction techniques, we propose a supervised method to enhance speech by means of finding a mapping function between noisy and clean speech signals based on deep neural networks (DNNs). In order to be able to handle a wide range of additive noises in real-world situations, a large training set that encompasses many possible combinations of speech and noise types, is first designed. A DNN architecture is then employed as a nonlinear regression function to ensure a powerful modeling capability. Several techniques have also been proposed to improve the DNN-based speech enhancement system, including global variance equalization to alleviate the over-smoothing problem of the regression model, and the dropout and noise-aware training strategies to further improve the generalization capability of DNNs to unseen noise conditions. Experimental results demonstrate that the proposed framework can achieve significant improvements in both objective and subjective measures over the conventional MMSE based technique. It is also interesting to observe that the proposed DNN approach can well suppress highly nonstationary noise, which is tough to handle in general. Furthermore, the resulting DNN model, trained with artificial synthesized data, is also effective in dealing with noisy speech data recorded in real-world scenarios without the generation of the annoying musical artifact commonly observed in conventional enhancement methods.

engineering, electrical & electronic,acoustics
Deep neural networks based speaker modeling at different levels of phonetic granularity

Yao Tian,Liang He,Meng Cai,Wei-Qiang Zhang,Jia Liu

DOI: https://doi.org/10.1109/icassp.2017.7953196

2017-01-01

Abstract:Recently, a hybrid deep neural network/i-vector framework has been proved effective for speaker verification, where the DNN trained to predict tied-triphone states (senones) is used to produce frame alignments for sufficient statistics extraction. In this work, in order to better understand the impact of different phonetic precision to speaker verification tasks, three levels of phonetic granularity are evaluated when doing frame alignments, which are tied-triphone state, monophone state and monophone. And the distribution of the features associated to a given phonetic unit is further modeled with multiple Gaussians rather than a single Gaussian. We also propose a fast and efficient way to generate phonetic units of different granularity by tying DNN's outputs according to the clustering results based on DNN derived senone embeddings. Experiments are carried out on the NIST SRE 2008 female tasks. Results show that using DNNs with less precise phonetic units and more Gaussians per phonetic unit for speaker modeling generalize better to different speaker verification tasks.
UnitNet-Based Hybrid Speech Synthesis

Xiao Zhou,Zhen-Hua Ling,Li-Rong Dai

DOI: https://doi.org/10.21437/interspeech.2021-1092

2021-01-01

Abstract:This paper presents a hybrid speech synthesis method based on UnitNet, a unified sequence-to-sequence (Seq2Seq) acoustic model for both statistical parametric speech synthesis (SPSS) and concatenative speech synthesis (CSS). This method combines CSS and SPSS approaches to synthesize different segments in an utterance. Comparing with the Tacotron2 model for Seq2Seq speech synthesis, UnitNet utilizes the phone boundaries of training data and its decoder contains autoregressive structures at both phone and frame levels. This hierarchical architecture can not only extract embedding vectors for representing phone-sized units in the corpus but also measure the dependency among consecutive units, which makes UnitNet capable of guiding the selection of phone-sized units for CSS. Furthermore, hybrid synthesis can be achieved by integrating the units generated by SPSS into the framework of CSS for the target phones without appropriate candidates in the corpus. Experimental results show that UnitNet can achieve comparable naturalness with Tacotron2 for SPSS and outperform our previous Tacotron-based method for CSS. Besides, the naturalness and inference efficiency of SPSS can be further improved through hybrid synthesis.
Hybrid Unit Model Based Non-uniform Unit Selection

NI Xin,CAI Lian-hong

DOI: https://doi.org/10.3969/j.issn.1000-1220.2005.06.047

2005-01-01

Abstract:This paper presents hybrid unit model based non-uniform unit selection in Chinese and English bilingual application oriented English text-to-speech system. In order to meet the speaker-consistent requirement and address the nonnative speaker problem which manifests deficiency and unrestraint in pronunciation, the Human Computer Speech Interaction Laboratory at Tsinghua University constructs acoustic inventory based upon hybrid unit model and implements an efficient, robust and flexible non-uniform unit selection method. All these prove effective experimentally in dealing with the nonnative speaker problem and in promoting the overall synthesis quality.
An Acoustic Model for English Speech Recognition Based on Deep Learning

Zhang Ling

DOI: https://doi.org/10.1109/icmtma.2019.00140

2019-04-01

Abstract:As one of the core modules of English speech recognition system, the performance of acoustic model directly affects the recognition effect of the final system. Aiming at the problems of acoustic modeling, model optimization and training efficiency in English speech recognition system, this paper focuses on the acoustic modeling method based on deep learning technology. By analyzing the basic theory of deep neural network and HMM, the structure and parameter configuration of DNN-HMM are intensively studied. A new hybrid network model is proposed by using clustered state instead of single factor state as the output unit of the neural network, and it is applied to the acoustic model of English speech recognition. The simulation results show that the improved scheme can get better recognition effect by increasing the enhancement of speech features under the condition of three-phoneme structure. In addition, the experimental results also verify that the acoustic modeling method based on DNN-HMM is superior to traditional GMM-HMM method.
A Unified Speaker-Dependent Speech Separation and Enhancement System Based on Deep Neural Networks.

Tian Gao,Jun Du,Li Xu,Cong Liu,Li-Rong Dai,Chin-Hui Lee

DOI: https://doi.org/10.1109/chinasip.2015.7230492

2015-01-01

Abstract:Speech enhancement and speech separation are important frontends of many speech processing systems. In real tasks, the background noises are often mixed with some human voice interferences. In this paper, we explore a framework to unify speech enhancement and speech separation for a speaker-dependent scenario based on deep neural networks (DNNs). Using a supervised method, DNN is adopted to directly model a nonlinear mapping function between noisy and clean speech signals. The signals of speaker interferers are considered as one type of universal noise signals in our framework. In order to be able to handle a wide range of additive noise in the real-world situations, a large training set that encompasses many possible combinations of speech and noise types, is designed. Experimental results demonstrate that the proposed framework can get the comparable performances to those single speech enhancement or separation systems. Furthermore, the resulting DNN model, trained with artificial synthesized data, is also effective in dealing with noisy speech data recorded in real-world conditions.

Learning and Modeling Unit Embeddings Using Deep Neural Networks for Unit-Selection-Based Mandarin Speech Synthesis

Learning and Modeling Unit Embeddings Using Deep Neural Networks for Unit-Selection-Based Mandarin Speech Synthesis.

Learning and Modeling Unit Embeddings for Improving HMM-based Unit Selection Speech Synthesis

Extracting Unit Embeddings Using Sequence-To-Sequence Acoustic Models for Unit Selection Speech Synthesis

DNN-based unit selection using frame-sized speech segments

Unit Selection Speech Synthesis Using Frame-Sized Speech Segments and Neural Network Based Acoustic Models

Unitnet: A Sequence-To-Sequence Acoustic Model For Concatenative Speech Synthesis

Selection of acoustic modeling unit for Tibetan speech recognition based on deep learning

Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition

A comparative study on selecting acoustic modeling units in deep neural networks based large vocabulary Chinese speech recognition

Statistical Acoustic Model Based Unit Selection Algorithm for Speech Synthesis

Deep Metric Learning For The Target Cost In Unit-Selection Speech Synthesizer

Trainable Unit Selection Speech Synthesis under Statistical Framework

State-Clustering Based Multiple Deep Neural Networks Modeling Approach for Speech Recognition

A regression approach to speech enhancement based on deep neural networks

Deep neural networks based speaker modeling at different levels of phonetic granularity

UnitNet-Based Hybrid Speech Synthesis

Hybrid Unit Model Based Non-uniform Unit Selection

An Acoustic Model for English Speech Recognition Based on Deep Learning

A Unified Speaker-Dependent Speech Separation and Enhancement System Based on Deep Neural Networks.