Abstract:State-of-the-art text-to-speech (TTS) synthesis models can produce monolingual speech with high intelligibility and naturalness. However, when the models are applied to synthesize code-switched (CS) speech, the performance declines seriously. Conventionally, developing a CS TTS system requires multilingual data to incorporate language-specific and cross-lingual knowledge. Recently, end-to-end (E2E) architecture has achieved satisfactory results in monolingual TTS. The architecture enables the training from one end of alphabetic text input to the other end of acoustic feature output. In this paper, we explore the use of E2E framework for CS TTS, using a combination of Mandarin and English monolingual speech corpus uttered by two female speakers. To handle alphabetic input from different languages, we explore two kinds of encoders: (1) shared multilingual encoder with explicit language embedding (LDE); (2) separated monolingual encoder (SPE) for each language. The two systems use identical decoder architecture, where a discriminative code is incorporated to enable the model to generate speech in one speaker's voice consistently. Experiments confirm the effectiveness of the proposed modifications on the E2E TTS framework in terms of quality and speaker similarity of the generated speech. Moreover, our proposed systems can generate controllable foreign-accented speech at character-level using only mixture of monolingual training data.

Hybrid Unit Model Based Non-uniform Unit Selection

Hierarchical Non-Uniform Unit Selection Based on Prosodic Structure

Statistical Acoustic Model Based Unit Selection Algorithm for Speech Synthesis

Selecting optimal non-uniform units for hierarchical unit selection

Stable boundary-based non-uniform unit selection in speech synthesis

A Novel Hybrid Mandarin Speech Synthesis System Using Different Base Units for Model Training and Concatenation

Trainable Unit Selection Speech Synthesis under Statistical Framework

Improved unit selection speech synthesis method utilizing subjective evaluation results on synthetic speech

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Unit Selection Speech Synthesis Integrating Automatic Error Detection

HMM-Based Hierarchical Unit Selection Combining Kullback-Leibler Divergence with Likelihood Criterion

A novel unit selection method for concatenation speech system using similarity measure

UnitNet-Based Hybrid Speech Synthesis

HMM-based Unit Selection Using F

Perceptual Clustering Based Unit Selection Optimization for Concatenative Text-to-speech Synthesis

Building HMM based unit-selection speech synthesis system using synthetic speech naturalness evaluation score

Optimization Method for Unit Selection Speech Synthesis Based on Synthesis Quality Predictions

HMM-based Unit Selection Using Frame Sized Speech Segments.

Mandarin-English Mixed Text to Speech Based on HCSIPA

HMM-BASED HIERARCHICALUNITSELECTIONCOMBINING KULLBACK-LEIBLER DIVERGENCE WITH LIKELIHOODCRITERION

A Decision Tree Based Approach for Construction of Speech Database and Unit Selection