Abstract:Accented text-to-speech (TTS) synthesis seeks to generate speech with an accent (L2) as a variant of the standard version (L1), which is challenging as L2 is different from L1 in terms of phonetic rendering and prosody pattern (pitch, energy, and duration variance, etc.). Accented TTS has several significant real-world applications, such as language learning, preserving and documenting endangered languages and dialects, etc. that make it an important area of research and development. Moreover, changing the accent intensity of any conversational AI system has the potential to allow specific users to understand its produced speech better. However, there is no intuitive solution for the control of the accent intensity for an utterance at both fine and coarse-grained levels, that are phoneme and utterance levels respectively. In this work, we propose a neural TTS architecture that allows us to control the accent style and its intensity. This is achieved through two novel mechanisms: 1) the front-end and back-end accent knowledge injection mechanism to enhance the accent interpretability of TTS modeling; and 2) an automatic speech recognition (ASR) based accent intensity modeling strategy to quantify the accent intensity in both L2 phoneme and utterance levels. In the front-end, a new accent variation adaptor seeks to project the accent-aware pitch, energy and duration features at a phoneme level, with the help of the fine-grained accent intensity information; In the back-end, a consistency constraint module that ensures the synthesized L2 speech manifests the expected accent intensity, is injected in the front-end, precisely. Experiments show that the proposed system attains superior performance to the baseline models in terms of accent rendering and intensity control. To our knowledge, this is the first study of accented TTS with explicit intensity control at both fine and coarse-grained levels.

Accented Text-to-Speech Synthesis with a Conditional Variational Autoencoder

Accent Conversion in Text-To-Speech Using Multi-Level VAE and Adversarial Training

DART: Disentanglement of Accent and Speaker Representation in Multispeaker Text-to-Speech

Controllable Accented Text-to-Speech Synthesis

Accented Text-to-Speech Synthesis with Limited Data

Accent conversion using discrete units with parallel data synthesized from controllable accented TTS

Controllable Accented Text-to-Speech Synthesis with Fine and Coarse-Grained Intensity Rendering

Multi-Scale Accent Modeling and Disentangling for Multi-Speaker Multi-Accent Text-to-Speech Synthesis

Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

Conditional Deep Hierarchical Variational Autoencoder for Voice Conversion

VANI: Very-lightweight Accent-controllable TTS for Native and Non-native speakers with Identity Preservation

Enhancing audio quality for expressive Neural Text-to-Speech

End-to-End Emotional Speech Synthesis Method Based on Conditional Variational Autoencoder

Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder

Modelling low-resource accents without accent-specific TTS frontend

Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis

Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencoder

Learning Latent Representations for Style Control and Transfer in End-to-end Speech Synthesis

Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech

Improving Pronunciation and Accent Conversion through Knowledge Distillation And Synthetic Ground-Truth from Native TTS