Abstract:Recently, two-pass end-to-end (E2E) automatic speech recognition (ASR) systems with the conformer model followed by a spelling correction backend have demonstrated remarkable progress and exceptional performance in general speech recognition tasks. However, these models may fail when they come to code-switching (CS) speech, where a speaker alternates words of two or more languages within a single sentence or across sentences. In this study, we propose a novel t ri-stage t raining two -pass (TripleT) E2E framework to improve the CS ASR performance by leveraging the individual attributes of each monolingual language. Our framework starts by introducing two symmetric language-specific encoders that are pre-trained using a large monolingual corpus. This improves the high-level acoustic representation of each individual language. Then, a bilingual acoustic learner (BAL) is proposed to combine these language-specific representations and transfer the monolingual acoustic attributes to code-switching properties. Next, these acoustic representations are further utilized to boost the spelling corrector by a context plus acoustic learner with the same structure as BAL. Finally, the whole proposed framework is fine-tuned using the CS corpus to achieve the final CS E2E ASR system. Our experiments are performed on a mixed training dataset consisting of 1000 hours of Mandarin data, 960 hours of English data, and 555.9 hours of Mandarin-English code-switching data. The ASR performances are evaluated on a 23.6 hours CS test set, and results show that our proposed TripleT-E2E framework achieves a 13.4% relative reduction in token error rate compared to a competitive two-pass E2E baseline model.

TALCS: An Open-Source Mandarin-English Code-Switching Corpus and a Speech Recognition Baseline

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Summary on the ISCSLP 2022 Chinese-English Code-Switching ASR Challenge

Non-autoregressive Mandarin-English Code-switching Speech Recognition

Exploring Retraining-Free Speech Recognition for Intra-sentential Code-Switching

Tri-stage training with language-specific encoder and bilingual acoustic learner for code-switching speech recognition

ATCSpeech: A Multilingual Pilot-Controller Speech Corpus from Real Air Traffic Control Environment

AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline

OC16-CE80: A Chinese-English Mixlingual Database and A Speech Recognition Baseline

WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition

End-to-End Mandarin Speech Recognition Combining CNN and BLSTM

MSR-86K: An Evolving, Multilingual Corpus with 86,300 Hours of Transcribed Audio for Speech Recognition Research

Transformer-Transducers for Code-Switched Speech Recognition

MERLIon CCS Challenge: A English-Mandarin code-switching child-directed speech corpus for language identification and diarization

Towards Language-Universal Mandarin-English Speech Recognition

IMUT-MC: a Speech Corpus for Mongolian Speech Recognition

Study of Semi-supervised Approaches to Improving English-Mandarin Code-Switching Speech Recognition

Design of Speech Corpus for Mandarin Text to Speech

Simple yet Effective Code-Switching Language Identification with Multitask Pre-Training and Transfer Learning

Language-specific Acoustic Boundary Learning for Mandarin-English Code-switching Speech Recognition

Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models