Abstract:Token-based text-to-speech (TTS) models have emerged as a promising avenue for generating natural and realistic speech, yet they grapple with low pronunciation accuracy, speaking style and timbre inconsistency, and a substantial need for diverse training data. In response, we introduce a novel hierarchical acoustic modeling approach complemented by a tailored data augmentation strategy and train it on the combination of real and synthetic data, scaling the data size up to 650k hours, leading to the zero-shot TTS model with 0.8B parameters. Specifically, our method incorporates a latent variable sequence containing supplementary acoustic information based on refined self-supervised learning (SSL) discrete units into the TTS model by a predictor. This significantly mitigates pronunciation errors and style mutations in synthesized speech. During training, we strategically replace and duplicate segments of the data to enhance timbre uniformity. Moreover, a pretrained few-shot voice conversion model is utilized to generate a plethora of voices with identical content yet varied timbres. This facilitates the explicit learning of utterance-level one-to-many mappings, enriching speech diversity and also ensuring consistency in timbre. Comparative experiments (Demo page: https://anonymous.4open.science/w/ham-tts/)demonstrate our model's superiority over VALL-E in pronunciation precision and maintaining speaking style, as well as timbre continuity.

Syllable HMM Based Mandarin TTS and Comparison with Concatenative TTS.

Comparison of Syllable/Phone HMM Based Mandarin TTS

A Novel Hybrid Mandarin Speech Synthesis System Using Different Base Units for Model Training and Concatenation

An Unified and Automatic Approach of Mandarin HTS System.

A Novel Hybrid Approach for Mandarin Speech Synthesis

Mandarin-English Mixed TTS Based on HCSIPA

An HMM-based Cantonese speech synthesis system

Syllable based DNN-HMM Cantonese Speech to Text System

HMM Based TTS for Mixed Language Text.

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

A unified sequence-to-sequence front-end model for Mandarin text-to-speech synthesis

A Novel Prosody Adaptation Method for Mandarin Concatenation-Based Text-to-speech System

Mandarin-English Mixed Text to Speech Based on HCSIPA

A Hierarchical Viterbi Algorithm For Mandarin Hybrid Speech Synthesis System

Label Transform Based Cross-Language Speaker Adaptation in Bilingual (Mandarin-English) TTS

HAM-TTS: Hierarchical Acoustic Modeling for Token-Based Zero-Shot Text-to-Speech with Model and Data Scaling

A comparable study of modeling units for end-to-end Mandarin speech recognition

Syllable Boundaries based Speech Segmentation in Demi-Syllable Level for Mandarin with HTK

Investigation of Modeling Units for Mandarin Speech Recognition Using Dfsmn-ctc-smbr

A novel unit selection method for concatenation speech system using similarity measure

Prosodic Modeling with Rich Syntactic Context in HMM-based Mandarin Speech Synthesis