Abstract:In recent decades, there has been a great deal of research into the problem of bilingual speech recognition - to develop a recognizer that can handle inter- and intra-sentential language switching between two languages. This paper presents our recent work on the development of a grammar-constrained, Mandarin-English bilingual Speech Recognition System (MESRS) for real world music retrieval. Two of the main difficult issues in handling the bilingual speech recognition systems for real world applications are tackled in this paper. One is to balance the performance and the complexity of the bilingual speech recognition system; the other is to effectively deal with the matrix language accents in embedded language**. In order to process the intra-sentential language switching and reduce the amount of data required to robustly estimate statistical models, a compact single set of bilingual acoustic models derived by phone set merging and clustering is developed instead of using two separate monolingual models for each language. In our study, a novel Two-pass phone clustering method based on Confusion Matrix (TCM) is presented and compared with the log-likelihood measure method. Experiments testify that TCM can achieve better performance. Since potential system users' native language is Mandarin which is regarded as a matrix language in our application, their pronunciations of English as the embedded language usually contain Mandarin accents. In order to deal with the matrix language accents in embedded language, different non-native adaptation approaches are investigated. Experiments show that model retraining method outperforms the other common adaptation methods such as Maximum A Posteriori (MAP). With the effective incorporation of approaches on phone clustering and non-native adaptation, the Phrase Error Rate (PER) of MESRS for English utterances was reduced by 24.47% relatively compared to the baseline monolingual English system while the PER on Mandarin utterances was comparable to that of the baseline monolingual Mandarin system. The performance for bilingual utterances achieved 22.37% relative PER reduction.

N-Best Re-scoring Approaches for Mandarin Speech Recognition

Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model

Improving Accented Mandarin Speech Recognition by Using Recurrent Neural Network Based Language Model Adaptation

Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking

Comparison of Non-native Speaker Adaptations for Large Vocabulary Continuous Mandarin Speech Recognition

A reweighting method for speech recognition with imbalanced data of Mandarin and sub-dialects

Recurrent Neural Network Based Language Model Adaptation for Accent Mandarin Speech.

Development of A Mandarin-English Bilingual Speech Recognition System for Real World Music Retrieval

Towards Language-Universal Mandarin-English Speech Recognition

Improvement Comparison of Different Lattice-based Discriminative Training Methods in Chinese-monolingual and Chinese-English-bilingual Speech Recognition

Effective Acoustic Modeling for Pronunciation Quality Scoring of Strongly Accented Mandarin Speech

Improving Blstm Rnn Based Mandarin Speech Recognition Using Accent Dependent Bottleneck Features

Research on speech recognition models in the Chinese dictation machine

Investigation of Deep Neural Network Acoustic Modelling Approaches for Low Resource Accented Mandarin Speech Recognition

Study of Semi-supervised Approaches to Improving English-Mandarin Code-Switching Speech Recognition

Recurrent Neural Network Language Model with Part-of-speech for Mandarin Speech Recognition.

Improved Mandarin Speech Recognition by Lattice Rescoring with Enhanced Tone Models

An Acoustic-Phonetic Analysis of Large Vocabulary Continuous Mandarin Speech Recognition for Non-Native Speakers

Improved Speech Recognition Using Discriminative Integration of Multiple Local Classifiers in Lattice Rescoring

Mandarin Speech Recognition for Nonnative Speakers Based on Pronunciation Dictionary Adaptation

Non-autoregressive Mandarin-English Code-switching Speech Recognition