Abstract:This study introduces a refined approach to Text-to-Speech (TTS) generation that significantly enhances sampling stability across languages, with a particular focus on Hebrew. By leveraging discrete semantic units with higher phonetic correlation obtained from a self-supervised model, our method addresses the inherent instability often encountered in TTS systems, especially those dealing with non-diacriticized scripts like Hebrew. Utilizing HuBERT codes, our model generates discrete representations that are optimized for TTS tasks, thereby reducing the dependency on diacritic-based text processing. This advancement not only simplifies the language modeling process but also improves the robustness and shows controllability of the speech output due to disentenglement properties of the semantic units. The inclusion of a speaker embedding in the vocoder further aids in capturing the unique vocal characteristics of the speaker, contributing to the naturalness of the synthesized speech. Our experimental results demonstrate that this approach not only maintains high performance in Hebrew but also shows adaptability to English, underscoring its effectiveness in enhancing stability in TTS systems universally. Our method, named LOTHM (Language of The Hebrew Man), outperforms existing methods in terms of stability while achieving naturalness and speaker similarity on par with previous methods, making it a compelling choice for future speech synthesis applications. Samples can be found in our page <a class="link-external link-http" href="http://pages.cs.huji.ac.il/adiyoss-lab/LoTHM" rel="external noopener nofollow">this http URL</a> .

A Language Modeling Approach to Diacritic-Free Hebrew TTS

Enhancing TTS Stability in Hebrew using Discrete Semantic Units

HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing

Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew

Nakdan: Professional Hebrew Diacritizer

Multilingual Sequence-to-Sequence Models for Hebrew NLP

D-Nikud: Enhancing Hebrew Diacritization with LSTM and Pretrained Models

Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities

MenakBERT -- Hebrew Diacriticizer

Restoring Hebrew Diacritics Without a Dictionary

A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration

On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models

Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis

HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew

ivrit.ai: A Comprehensive Dataset of Hebrew Speech for AI Research and Development

DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew

Towards Zero-Shot Text-To-Speech for Arabic Dialects

What's Wrong with Hebrew NLP? And How to Make it Right

Effective Deep Learning Models for Automatic Diacritization of Arabic Text

HAM-TTS: Hierarchical Acoustic Modeling for Token-Based Zero-Shot Text-to-Speech with Model and Data Scaling

Prosody-controllable spontaneous TTS with neural HMMs