Abstract:There exist three approaches for multilingual and crosslingual automatic speech recognition (MCL-ASR) - supervised pre-training with phonetic or graphemic transcription, and self-supervised pre-training. We find that pre-training with phonetic supervision has been underappreciated so far for MCL-ASR, while conceptually it is more advantageous for information sharing between different languages. This paper explores the approach of pre-training with weakly phonetic supervision towards data-efficient MCL-ASR, which is called Whistle. We relax the requirement of gold-standard human-validated phonetic transcripts, and obtain International Phonetic Alphabet (IPA) based transcription by leveraging the LanguageNet grapheme-to-phoneme (G2P) models. We construct a common experimental setup based on the CommonVoice dataset, called CV-Lang10, with 10 seen languages and 2 unseen languages. A set of experiments are conducted on CV-Lang10 to compare, as fair as possible, the three approaches under the common setup for MCL-ASR. Experiments demonstrate the advantages of phoneme-based models (Whistle) for MCL-ASR, in terms of speech recognition for seen languages, crosslingual performance for unseen languages with different amounts of few-shot data, overcoming catastrophic forgetting, and training <a class="link-external link-http" href="http://efficiency.It" rel="external noopener nofollow">this http URL</a> is found that when training data is more limited, phoneme supervision can achieve better results compared to subword supervision and self-supervision, thereby providing higher data-efficiency. To support reproducibility and promote future research along this direction, we will release the code, models and data for the whole pipeline of Whistle at <a class="link-external link-https" href="https://github.com/thu-spmi/CAT" rel="external noopener nofollow">this https URL</a> upon publication.

WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words

Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings

Acquiring Linguistic Knowledge from Multimodal Input

Efficient Compression of Multitask Multilingual Speech Models

Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts

TextMI: Textualize Multimodal Information for Integrating Non-verbal Cues in Pre-trained Language Models

Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing

Multimodal Audio-textual Architecture for Robust Spoken Language Understanding

Multimodal Pretraining from Monolingual to Multilingual

WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding

Multimodal Speech Recognition for Language-Guided Embodied Agents

Language as the Medium: Multimodal Video Classification through text only

Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation

Visatronic: A Multimodal Decoder-Only Model for Speech Synthesis

Advancing Audio Emotion and Intent Recognition with Large Pre-Trained Models and Bayesian Inference

Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model

Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision

SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data

CM-BERT

VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning

SpeechVerse: A Large-scale Generalizable Audio Language Model