Abstract:The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing.

Concept-to-Speech Generation with Knowledge Sharing for Acoustic Modelling and Utterance Filtering

Concept-to-Speech generation with knowledge sharing for acoustic modelling and utterance filtering

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Prosody Modelling with Pre-trained Cross-utterance Representations for Improved Speech Synthesis

M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesis

Knowledge-based Linguistic Encoding for End-to-End Mandarin Text-to-Speech Synthesis

UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding

An Unified and Automatic Approach of Mandarin HTS System.

Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling

An Enhanced Text Analysis Approach in Text-to-Speech Synthesis for Mandarin Chinese

Hierarchical Cross-Modality Knowledge Transfer with Sinkhorn Attention for CTC-based ASR

Improving Prosody with Linguistic and Bert Derived Features in Multi-Speaker Based Mandarin Chinese Neural TTS

Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesis

A context-aware knowledge transferring strategy for CTC-based ASR

FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis

Modeling Prosody Patterns for Chinese Expressive Text-to-speech Synthesis

Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information

An HMM-based Cantonese speech synthesis system

Multi-Task Learning for Prosodic Structure Generation Using BLSTM RNN with Structured Output Layer

Enhancing CTC-based speech recognition with diverse modeling units

Learning Prosodic Patterns for Mandarin Speech Synthesis