Abstract:This paper presents a novel multistage fine-tuning strategy designed to enhance automatic speech recognition (ASR) performance in low-resource languages using OpenAI's Whisper model. In this approach we aim to build ASR model for languages with limited digital resources by sequentially adapting the model across linguistically similar languages. We experimented this on the Malasar language, a Dravidian language spoken by approximately ten thousand people in the Western Ghats of South India. Malasar language faces critical challenges for technological intervention due to its lack of a native script and absence of digital or spoken data resources. Working in collaboration with Wycliffe India and Malasar community members, we created a spoken Malasar corpus paired with transcription in Tamil script, a closely related major language. In our approach to build ASR model for Malasar, we first build an intermediate Tamil ASR, leveraging higher data availability for Tamil annotated speech. This intermediate model is subsequently fine-tuned on Malasar data, allowing for more effective ASR adaptation despite limited resources. The multistage fine-tuning strategy demonstrated significant improvements over direct fine-tuning on Malasar data alone, achieving a word error rate (WER) of 51.9%, which is 4.5% absolute reduction when compared to the direct fine-tuning method. Further a WER reduction to 47.3% was achieved through punctuation removal in post-processing, which addresses formatting inconsistencies that impact evaluation. Our results underscore the effectiveness of sequential multistage fine-tuning combined with targeted post-processing as a scalable strategy for ASR system development in low-resource languages, especially where linguistic similarities can be leveraged to bridge gaps in training data.

Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognition

Optimizing Data Usage for Low-Resource Speech Recognition

Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages

Efficient Compression of Multitask Multilingual Speech Models

Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling

Exploring Effective Data Utilization for Low-Resource Speech Recognition

Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts

LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR

Multilingual Meta-Transfer Learning for Low-Resource Speech Recognition

Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper

Cross lingual transfer learning for zero-resource domain adaptation

Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translation

Meta-Whisper: Speech-Based Meta-ICL for ASR on Low-Resource Languages

Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text

Hierarchical Softmax for End-to-End Low-resource Multilingual Speech Recognition

Multistage Fine-tuning Strategies for Automatic Speech Recognition in Low-resource Languages

Universal Cross-Lingual Data Generation for Low Resource ASR

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

Multilingual Recurrent Neural Networks with Residual Learning for Low-Resource Speech Recognition.

A General Procedure for Improving Language Models in Low-Resource Speech Recognition

Exploration of Whisper fine-tuning strategies for low-resource ASR