Abstract:The end-to-end ASR model is often desired in the streaming multilingual scenario since it is easier to deploy and can benefit from pre-trained speech models such as powerful foundation models. Meanwhile, the heterogeneous nature and imbalanced data abundance of different languages may cause performance degradation, leading to asynchronous peak performance for different languages during training, especially on tail ones. Sometimes even the data itself may become unavailable as a result of the enhanced privacy protection. Existing work tend to significantly increase the model size or learn language-specific decoders to accommodate each language separately. In this study, we explore simple yet effective Language-Dependent Adapter (LDA) finetuning under a cascaded Conformer transducer framework enhanced by teacher pseudo-labeling for tail languages in the streaming multilingual ASR. The adapter only accounts for 0.4% of the full model per language. It is plugged into the frozen foundation model and is the only trainable module during the finetuning process with noisy student training. The final model merges the adapter parameters from different checkpoints for different languages. The model performance is validated on a challenging multilingual dictation dataset, which includes 39 tail languages across Latin, Greek, Arabic, etc. Our proposed method brings 12.2% word error rate reduction on average and up to 37.5% on a single locale. Furthermore, we show that our parameter-efficient LDA can match the quality of the full model finetuning, thus greatly alleviating the asynchronous peak performance issue.

An Adapter-Based Unified Model for Multiple Spoken Language Processing Tasks

ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks

Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification

Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models

Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition

Audio-AdapterFusion: A Task-ID-free Approach for Efficient and Non-Destructive Multi-task Speech Recognition

Efficient Adapters for Giant Speech Models

Exploring efficient-tuning methods in self-supervised speech models

ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptation

Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding

Don't Stop Self-Supervision: Accent Adaptation of Speech Representations via Residual Adapters

Enhancing Two-Stage Finetuning for Speech Emotion Recognition Using Adapters.

Lightweight Adapter Tuning for Multilingual Speech Translation

Adapting the adapters for code-switching in multilingual ASR

Adapting Multi-Lingual ASR Models for Handling Multiple Talkers

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

A Unified Speaker Adaptation Method for Speech Synthesis using Transcribed and Untranscribed Speech with Backpropagation

UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASR

Efficient Transfer Learning Methods Using Parameterization in Few-Shot Speech Recognition

One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization