Abstract:There exist three approaches for multilingual and crosslingual automatic speech recognition (MCL-ASR) - supervised pre-training with phonetic or graphemic transcription, and self-supervised pre-training. We find that pre-training with phonetic supervision has been underappreciated so far for MCL-ASR, while conceptually it is more advantageous for information sharing between different languages. This paper explores the approach of pre-training with weakly phonetic supervision towards data-efficient MCL-ASR, which is called Whistle. We relax the requirement of gold-standard human-validated phonetic transcripts, and obtain International Phonetic Alphabet (IPA) based transcription by leveraging the LanguageNet grapheme-to-phoneme (G2P) models. We construct a common experimental setup based on the CommonVoice dataset, called CV-Lang10, with 10 seen languages and 2 unseen languages. A set of experiments are conducted on CV-Lang10 to compare, as fair as possible, the three approaches under the common setup for MCL-ASR. Experiments demonstrate the advantages of phoneme-based models (Whistle) for MCL-ASR, in terms of speech recognition for seen languages, crosslingual performance for unseen languages with different amounts of few-shot data, overcoming catastrophic forgetting, and training <a class="link-external link-http" href="http://efficiency.It" rel="external noopener nofollow">this http URL</a> is found that when training data is more limited, phoneme supervision can achieve better results compared to subword supervision and self-supervision, thereby providing higher data-efficiency. To support reproducibility and promote future research along this direction, we will release the code, models and data for the whole pipeline of Whistle at <a class="link-external link-https" href="https://github.com/thu-spmi/CAT" rel="external noopener nofollow">this https URL</a> upon publication.

Simple and Effective Zero-shot Cross-lingual Phoneme Recognition

Cross-Lingual Self-training to Learn Multilingual Representation for Low-Resource Speech Recognition

Multilingual Speech Translation with Efficient Finetuning of Pretrained Models

Scaling A Simple Approach to Zero-Shot Speech Recognition

Improved acoustic word embeddings for zero-resource languages using multilingual transfer

Multilingual acoustic word embedding models for processing zero-resource languages

Cross-Lingual Transfer Learning for Speech Translation

Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning.

Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASR

Towards Zero-Shot Code-Switched Speech Recognition

Language-Driven Cross-Modal Classifier for Zero-Shot Multi-Label Image Recognition

Language-Universal Speech Attributes Modeling for Zero-Shot Multilingual Spoken Keyword Recognition

Cross lingual transfer learning for zero-resource domain adaptation

Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation.

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Improving Automatic Speech Recognition Performance for Low-Resource Languages With Self-Supervised Models

Learning Cross-lingual Visual Speech Representations

Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer

Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translation

Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining