Abstract:Benefiting from massive and diverse data sources, speech foundation models exhibit strong generalization and knowledge transfer capabilities to a wide range of downstream tasks. However, a limitation arises from their exclusive handling of single-speaker speech input, making them ineffective in recognizing multi-speaker overlapped speech, a common occurrence in real-world scenarios. In this study, we delve into the adaptation of speech foundation models to eliminate interfering speakers from overlapping speech and perform target-speaker automatic speech recognition (TS-ASR). Initially, we utilize the Whisper model as the foundation for adaptation and conduct a thorough comparison of its integration with existing target-speaker adaptation techniques. We then propose an innovative model termed Speaker-Querying Whisper (SQ-Whisper), which employs a set number of trainable queries to capture speaker prompts from overlapping speech based on target-speaker enrollment. These prompts serve to steer the model in extracting speaker-specific features and accurately recognizing target-speaker transcriptions. Experimental results demonstrate that our approach effectively adapts the pre-trained speech foundation model to TS-ASR. Compared with the robust TS-HuBERT model, the proposed SQ-Whisper significantly improves performance, yielding up to 15% and 10% relative reductions in word error rates (WERs) on the Libri2Mix and WSJ0-2Mix datasets, respectively. With data augmentation, we establish new state-of-the-art WERs of 14.6% on the Libri2Mix Test set and 4.4% on the WSJ0-2Mix Test set. Furthermore, we evaluate our model on the real-world AMI meeting dataset, which shows consistent improvement over other adaptation methods.

LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR

Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper

Dual-Pipeline with Low-Rank Adaptation for New Language Integration in Multilingual ASR

A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenario

Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts

M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper

Sparsely Shared LoRA on Whisper for Child Speech Recognition

Efficient Compression of Multitask Multilingual Speech Models

PI-Whisper: An Adaptive and Incremental ASR Framework for Diverse and Evolving Speaker Characteristics

Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling

Efficient Adaptation of Multilingual Models for Japanese ASR

Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages

SQ-Whisper: Speaker-Querying based Whisper Model for Target-Speaker ASR

Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text

Implementation of a Whisper Architecture-Based Turkish Automatic Speech Recognition (ASR) System and Evaluation of the Effect of Fine-Tuning with a Low-Rank Adaptation (LoRA) Adapter on Its Performance

Exploring Native and Non-Native English Child Speech Recognition With Whisper

Meta-Whisper: Speech-Based Meta-ICL for ASR on Low-Resource Languages

Cross-Lingual Transfer Learning for Speech Translation

A Parameter-efficient Language Extension Framework for Multilingual ASR

Using fine-tuning and min lookahead beam search to improve Whisper

Real-time multilingual speech recognition and speaker diarization system based on Whisper segmentation