Abstract:There exist three approaches for multilingual and crosslingual automatic speech recognition (MCL-ASR) - supervised pre-training with phonetic or graphemic transcription, and self-supervised pre-training. We find that pre-training with phonetic supervision has been underappreciated so far for MCL-ASR, while conceptually it is more advantageous for information sharing between different languages. This paper explores the approach of pre-training with weakly phonetic supervision towards data-efficient MCL-ASR, which is called Whistle. We relax the requirement of gold-standard human-validated phonetic transcripts, and obtain International Phonetic Alphabet (IPA) based transcription by leveraging the LanguageNet grapheme-to-phoneme (G2P) models. We construct a common experimental setup based on the CommonVoice dataset, called CV-Lang10, with 10 seen languages and 2 unseen languages. A set of experiments are conducted on CV-Lang10 to compare, as fair as possible, the three approaches under the common setup for MCL-ASR. Experiments demonstrate the advantages of phoneme-based models (Whistle) for MCL-ASR, in terms of speech recognition for seen languages, crosslingual performance for unseen languages with different amounts of few-shot data, overcoming catastrophic forgetting, and training <a class="link-external link-http" href="http://efficiency.It" rel="external noopener nofollow">this http URL</a> is found that when training data is more limited, phoneme supervision can achieve better results compared to subword supervision and self-supervision, thereby providing higher data-efficiency. To support reproducibility and promote future research along this direction, we will release the code, models and data for the whole pipeline of Whistle at <a class="link-external link-https" href="https://github.com/thu-spmi/CAT" rel="external noopener nofollow">this https URL</a> upon publication.

Reading Miscue Detection in Primary School through Automatic Speech Recognition

Automatic Assessment of Oral Reading Accuracy for Reading Diagnostics

An ASR-Based Tutor for Learning to Read: How to Optimize Feedback to First Graders

Automatic Speech Recognition of Non-Native Child Speech for Language Learning Applications

Error-preserving Automatic Speech Recognition of Young English Learners' Language

Kid-Whisper: Towards Bridging the Performance Gap in Automatic Speech Recognition for Children VS. Adults

Exploring Native and Non-Native English Child Speech Recognition With Whisper

Experiments of ASR-based mispronunciation detection for children and adult English learners

Using Open-Source Automatic Speech Recognition Tools for the Annotation of Dutch Infant-Directed Speech

Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision

Automatic speech recognition (ASR) for the diagnosis of pronunciation of speech sound disorders in Korean children

Non-Native Children's Automatic Speech Recognition: the INTERSPEECH 2020 Shared Task ALTA Systems.

Predicting lexical skills from oral reading with acoustic measures

Adapting an ASR Foundation Model for Spoken Language Assessment

Adaptive Frequency Cepstral Coefficients for Word Mispronunciation Detection

Automatic recognition and detection of aphasic natural speech

Development of speechreading supplements based on automatic speech recognition

An LVCSR Based Automatic Scoring Method in English Reading Tests

Adaptation of Whisper models to child speech recognition

Non-Intrusive Speech Intelligibility Prediction for Hearing-Impaired Users using Intermediate ASR Features and Human Memory Models

Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition