Abstract:Recently, two-pass end-to-end (E2E) automatic speech recognition (ASR) systems with the conformer model followed by a spelling correction backend have demonstrated remarkable progress and exceptional performance in general speech recognition tasks. However, these models may fail when they come to code-switching (CS) speech, where a speaker alternates words of two or more languages within a single sentence or across sentences. In this study, we propose a novel t ri-stage t raining two -pass (TripleT) E2E framework to improve the CS ASR performance by leveraging the individual attributes of each monolingual language. Our framework starts by introducing two symmetric language-specific encoders that are pre-trained using a large monolingual corpus. This improves the high-level acoustic representation of each individual language. Then, a bilingual acoustic learner (BAL) is proposed to combine these language-specific representations and transfer the monolingual acoustic attributes to code-switching properties. Next, these acoustic representations are further utilized to boost the spelling corrector by a context plus acoustic learner with the same structure as BAL. Finally, the whole proposed framework is fine-tuned using the CS corpus to achieve the final CS E2E ASR system. Our experiments are performed on a mixed training dataset consisting of 1000 hours of Mandarin data, 960 hours of English data, and 555.9 hours of Mandarin-English code-switching data. The ASR performances are evaluated on a 23.6 hours CS test set, and results show that our proposed TripleT-E2E framework achieves a 13.4% relative reduction in token error rate compared to a competitive two-pass E2E baseline model.

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Investigation of Effectively Synthesizing Code-Switched Speech Using Highly Imbalanced Mix-Lingual Data

Learning Language and Speaker Information for Code-Switch Speech Synthesis with Limited Data.

Text Enhancement for Paragraph Processing in End-to-End Code-switching TTS

Tri-stage training with language-specific encoder and bilingual acoustic learner for code-switching speech recognition

Knowledge-based Linguistic Encoding for End-to-End Mandarin Text-to-Speech Synthesis

Code-Switching Text Generation and Injection in Mandarin-English ASR

Efficient Decoding Self-Attention for End-to-end Speech Synthesis

Code-switched speech synthesis using bilingual phonetic posteriorgram with only monolingual corpora

End-to-End Speech Translation for Code Switched Speech

A Cross-Language State Mapping Approach to Bilingual (Mandarin-English) TTS

An End-to-End Speech Recognition System Based on Shared Encoder

Improve Bilingual TTS Using Dynamic Language and Phonology Embedding

Emphatic Speech Synthesis and Control Based on Characteristic Transferring in End-to-End Speech Synthesis

Conversational End-to-End TTS for Voice Agent

An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement

End-to-End Mongolian Text-to-Speech System

End-to-End Code-Switching ASR for Low-Resourced Language Pairs

Cross-lingual Multispeaker Text-to-Speech under Limited-Data Scenario

HMM Based TTS for Mixed Language Text.

M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesis