Abstract:State-of-the-art text-to-speech (TTS) synthesis models can produce monolingual speech with high intelligibility and naturalness. However, when the models are applied to synthesize code-switched (CS) speech, the performance declines seriously. Conventionally, developing a CS TTS system requires multilingual data to incorporate language-specific and cross-lingual knowledge. Recently, end-to-end (E2E) architecture has achieved satisfactory results in monolingual TTS. The architecture enables the training from one end of alphabetic text input to the other end of acoustic feature output. In this paper, we explore the use of E2E framework for CS TTS, using a combination of Mandarin and English monolingual speech corpus uttered by two female speakers. To handle alphabetic input from different languages, we explore two kinds of encoders: (1) shared multilingual encoder with explicit language embedding (LDE); (2) separated monolingual encoder (SPE) for each language. The two systems use identical decoder architecture, where a discriminative code is incorporated to enable the model to generate speech in one speaker's voice consistently. Experiments confirm the effectiveness of the proposed modifications on the E2E TTS framework in terms of quality and speaker similarity of the generated speech. Moreover, our proposed systems can generate controllable foreign-accented speech at character-level using only mixture of monolingual training data.

AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation.

Pre-Trained Acoustic-and-Textual Modeling for End-To-End Speech-To-Text Translation.

Bridging the Modality Gap for Speech-to-Text Translation

Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding.

Efficient Decoding Self-Attention for End-to-end Speech Synthesis

Towards End-to-end Speech-to-text Translation with Two-pass Decoding

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Improving Automatic Speech Recognition and Speech Translation Via Word Embedding Prediction

Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks

Worse WER, but Better BLEU? Leveraging Word Embedding As Intermediate in Multitask End-to-End Speech Translation

Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders

Dynamic Speaker Representations Adjustment and Decoder Factorization for Speaker Adaptation in End-to-End Speech Synthesis

Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation

Duplex Diffusion Models Improve Speech-to-Speech Translation

Improving Speech Translation by Understanding the Speech From Latent Code

AlloST: Low-resource Speech Translation without Source Transcription

Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data.

Learning Adaptive Segmentation Policy for End-to-End Simultaneous Translation

Non-Parametric Domain Adaptation for End-to-End Speech Translation

Coupling Speech Encoders with Downstream Text Models

R-BI: Regularized Batched Inputs enhance Incremental Decoding Framework for Low-Latency Simultaneous Speech Translation