Abstract:Benefiting from massive and diverse data sources, speech foundation models exhibit strong generalization and knowledge transfer capabilities to a wide range of downstream tasks. However, a limitation arises from their exclusive handling of single-speaker speech input, making them ineffective in recognizing multi-speaker overlapped speech, a common occurrence in real-world scenarios. In this study, we delve into the adaptation of speech foundation models to eliminate interfering speakers from overlapping speech and perform target-speaker automatic speech recognition (TS-ASR). Initially, we utilize the Whisper model as the foundation for adaptation and conduct a thorough comparison of its integration with existing target-speaker adaptation techniques. We then propose an innovative model termed Speaker-Querying Whisper (SQ-Whisper), which employs a set number of trainable queries to capture speaker prompts from overlapping speech based on target-speaker enrollment. These prompts serve to steer the model in extracting speaker-specific features and accurately recognizing target-speaker transcriptions. Experimental results demonstrate that our approach effectively adapts the pre-trained speech foundation model to TS-ASR. Compared with the robust TS-HuBERT model, the proposed SQ-Whisper significantly improves performance, yielding up to 15% and 10% relative reductions in word error rates (WERs) on the Libri2Mix and WSJ0-2Mix datasets, respectively. With data augmentation, we establish new state-of-the-art WERs of 14.6% on the Libri2Mix Test set and 4.4% on the WSJ0-2Mix Test set. Furthermore, we evaluate our model on the real-world AMI meeting dataset, which shows consistent improvement over other adaptation methods.

Enhancing Whisper Model for Pronunciation Assessment with Multi-Adapters

A Study on Incorporating Whisper for Robust Speech Assessment

Adapting an ASR Foundation Model for Spoken Language Assessment

An Mandarin Pronunciation Quality Assessment System Using Two Kinds of Acoustic Models

Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

Effective Acoustic Modeling for Pronunciation Quality Scoring of Strongly Accented Mandarin Speech

Improving Whispered Speech Recognition Performance using Pseudo-whispered based Data Augmentation

Pronunciation Assessment with Multi-modal Large Language Models

End-to-End Word-Level Pronunciation Assessment with MASK Pre-training

MultiPA: A Multi-task Speech Pronunciation Assessment Model for Open Response Scenarios

M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper

SQ-Whisper: Speaker-Querying based Whisper Model for Target-Speaker ASR

Perceiver-Prompt: Flexible Speaker Adaptation in Whisper for Chinese Disordered Speech Recognition

Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding

Preserving Phonemic Distinctions for Ordinal Regression: A Novel Loss Function for Automatic Pronunciation Assessment

Disentangling the Contribution of Non-native Speech in Automated Pronunciation Assessment

A Hierarchical Context-aware Modeling Approach for Multi-aspect and Multi-granular Pronunciation Assessment

Auto Adapted English Pronunciation Evaluation: A Fuzzy Integral Approach

Whisper-PMFA: Partial Multi-Scale Feature Aggregation for Speaker Verification using Whisper Models

Acoustic Model Refining Algorithm for Pronunciation Quality Evaluation

Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text