Abstract:Benefiting from massive and diverse data sources, speech foundation models exhibit strong generalization and knowledge transfer capabilities to a wide range of downstream tasks. However, a limitation arises from their exclusive handling of single-speaker speech input, making them ineffective in recognizing multi-speaker overlapped speech, a common occurrence in real-world scenarios. In this study, we delve into the adaptation of speech foundation models to eliminate interfering speakers from overlapping speech and perform target-speaker automatic speech recognition (TS-ASR). Initially, we utilize the Whisper model as the foundation for adaptation and conduct a thorough comparison of its integration with existing target-speaker adaptation techniques. We then propose an innovative model termed Speaker-Querying Whisper (SQ-Whisper), which employs a set number of trainable queries to capture speaker prompts from overlapping speech based on target-speaker enrollment. These prompts serve to steer the model in extracting speaker-specific features and accurately recognizing target-speaker transcriptions. Experimental results demonstrate that our approach effectively adapts the pre-trained speech foundation model to TS-ASR. Compared with the robust TS-HuBERT model, the proposed SQ-Whisper significantly improves performance, yielding up to 15% and 10% relative reductions in word error rates (WERs) on the Libri2Mix and WSJ0-2Mix datasets, respectively. With data augmentation, we establish new state-of-the-art WERs of 14.6% on the Libri2Mix Test set and 4.4% on the WSJ0-2Mix Test set. Furthermore, we evaluate our model on the real-world AMI meeting dataset, which shows consistent improvement over other adaptation methods.

Using fine-tuning and min lookahead beam search to improve Whisper

Exploration of Whisper fine-tuning strategies for low-resource ASR

Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection

Efficient Compression of Multitask Multilingual Speech Models

Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text

Extending Whisper with prompt tuning to target-speaker ASR

Meta-Prompt: Boosting Whisper's Performance in Low-Resource Speech Recognition

DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognition

Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts

Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper

LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR

Improving Whispered Speech Recognition Performance using Pseudo-whispered based Data Augmentation

Whisper-SV: Adapting Whisper for Low-data-resource Speaker Verification

SQ-Whisper: Speaker-Querying based Whisper Model for Target-Speaker ASR

Exploring Native and Non-Native English Child Speech Recognition With Whisper

Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling

Sparsely Shared LoRA on Whisper for Child Speech Recognition

A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting

Transfer Learning from Whisper for Microscopic Intelligibility Prediction

Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language Understanding

Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR