Abstract:Recently, two-pass end-to-end (E2E) automatic speech recognition (ASR) systems with the conformer model followed by a spelling correction backend have demonstrated remarkable progress and exceptional performance in general speech recognition tasks. However, these models may fail when they come to code-switching (CS) speech, where a speaker alternates words of two or more languages within a single sentence or across sentences. In this study, we propose a novel t ri-stage t raining two -pass (TripleT) E2E framework to improve the CS ASR performance by leveraging the individual attributes of each monolingual language. Our framework starts by introducing two symmetric language-specific encoders that are pre-trained using a large monolingual corpus. This improves the high-level acoustic representation of each individual language. Then, a bilingual acoustic learner (BAL) is proposed to combine these language-specific representations and transfer the monolingual acoustic attributes to code-switching properties. Next, these acoustic representations are further utilized to boost the spelling corrector by a context plus acoustic learner with the same structure as BAL. Finally, the whole proposed framework is fine-tuned using the CS corpus to achieve the final CS E2E ASR system. Our experiments are performed on a mixed training dataset consisting of 1000 hours of Mandarin data, 960 hours of English data, and 555.9 hours of Mandarin-English code-switching data. The ASR performances are evaluated on a 23.6 hours CS test set, and results show that our proposed TripleT-E2E framework achieves a 13.4% relative reduction in token error rate compared to a competitive two-pass E2E baseline model.

Improving Transformer Based End-to-End Code-Switching Speech Recognition Using Language Identification

Transformer-Transducers for Code-Switched Speech Recognition

Rnn-transducer with Language Bias for End-to-end Mandarin-English Code-switching Speech Recognition

Simple yet Effective Code-Switching Language Identification with Multitask Pre-Training and Transfer Learning

Language-specific Acoustic Boundary Learning for Mandarin-English Code-switching Speech Recognition

Leveraging Language ID to Calculate Intermediate CTC Loss for Enhanced Code-Switching Speech Recognition

Tri-stage training with language-specific encoder and bilingual acoustic learner for code-switching speech recognition

Non-autoregressive Mandarin-English Code-switching Speech Recognition

Enhancing Code-Switching ASR Leveraging Non-Peaky CTC Loss and Deep Language Posterior Injection

Automatic Spelling Correction with Transformer for CTC-based End-to-End Speech Recognition

Exploring Retraining-Free Speech Recognition for Intra-sentential Code-Switching

Reducing Multilingual Context Confusion for End-to-end Code-switching Automatic Speech Recognition

An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement

Minimum word error training for non-autoregressive Transformer-based code-switching ASR

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Decoupling Pronunciation and Language for End-to-End Code-Switching Automatic Speech Recognition.

Attention-Guided Adaptation for Code-Switching Speech Recognition

Reducing language context confusion for end-to-end code-switching automatic speech recognition

Investigation of Transformer Based Spelling Correction Model for CTC-based End-to-End Mandarin Speech Recognition

Improving RNN Transducer Based ASR with Auxiliary Tasks

An improved hybrid CTC-Attention model for speech recognition