Abstract:Recently, two-pass end-to-end (E2E) automatic speech recognition (ASR) systems with the conformer model followed by a spelling correction backend have demonstrated remarkable progress and exceptional performance in general speech recognition tasks. However, these models may fail when they come to code-switching (CS) speech, where a speaker alternates words of two or more languages within a single sentence or across sentences. In this study, we propose a novel t ri-stage t raining two -pass (TripleT) E2E framework to improve the CS ASR performance by leveraging the individual attributes of each monolingual language. Our framework starts by introducing two symmetric language-specific encoders that are pre-trained using a large monolingual corpus. This improves the high-level acoustic representation of each individual language. Then, a bilingual acoustic learner (BAL) is proposed to combine these language-specific representations and transfer the monolingual acoustic attributes to code-switching properties. Next, these acoustic representations are further utilized to boost the spelling corrector by a context plus acoustic learner with the same structure as BAL. Finally, the whole proposed framework is fine-tuned using the CS corpus to achieve the final CS E2E ASR system. Our experiments are performed on a mixed training dataset consisting of 1000 hours of Mandarin data, 960 hours of English data, and 555.9 hours of Mandarin-English code-switching data. The ASR performances are evaluated on a 23.6 hours CS test set, and results show that our proposed TripleT-E2E framework achieves a 13.4% relative reduction in token error rate compared to a competitive two-pass E2E baseline model.

Code-Switching Text Generation and Injection in Mandarin-English ASR

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Data Augmentation for End-to-end Code-switching Speech Recognition

Enhancing Code-Switching ASR Leveraging Non-Peaky CTC Loss and Deep Language Posterior Injection

Transformer-Transducers for Code-Switched Speech Recognition

Improving Multilingual and Code-Switching ASR Using Large Language Model Generated Text

Non-autoregressive Mandarin-English Code-switching Speech Recognition

Data Augmentation for Code-Switch Language Modeling by Fusing Multiple Text Generation Methods.

Improving Low Resource Code-switched ASR using Augmented Code-switched TTS

An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement

Learning Language and Speaker Information for Code-Switch Speech Synthesis with Limited Data.

Minimum word error training for non-autoregressive Transformer-based code-switching ASR

Code-switching Sentence Generation by Generative Adversarial Networks and its Application to Data Augmentation

Investigation of Effectively Synthesizing Code-Switched Speech Using Highly Imbalanced Mix-Lingual Data

Bi-Encoder Transformer Network for Mandarin-English Code-Switching Speech Recognition Using Mixture of Experts.

Text Injection for Capitalization and Turn-Taking Prediction in Speech Models

Tri-stage training with language-specific encoder and bilingual acoustic learner for code-switching speech recognition

Decoupling Pronunciation and Language for End-to-End Code-Switching Automatic Speech Recognition.

Efficient Decoding Self-Attention for End-to-end Speech Synthesis

Pronunciation Generation for Foreign Language Words in Intra-Sentential Code-Switching Speech Recognition

Text Enhancement for Paragraph Processing in End-to-End Code-switching TTS