Abstract:Self-supervised learning (SSL) based speech pre-training has attracted much attention for its capability of extracting rich representations learned from massive unlabeled data. On the other hand, the use of weakly-supervised data is less explored for speech pre-training. To fill this gap, we propose a weakly-supervised speech pre-training method based on speaker-aware speech data. It adopts a similar training procedure to the widely-used masked speech prediction based SSL framework, while incorporating additional target-speaker enrollment information as an auxiliary input. In this way, the learned representation is steered towards the target speaker even in the presence of highly overlapping interference, allowing potential applications to tasks such as target speech recognition. Our experiments on Libri2Mix and WSJ0-2mix datasets show that the proposed model achieves significantly better ASR performance compared to WavLM, the state-of-the-art SSL model with denoising capability.

What problem does this paper attempt to address?

The paper attempts to address the problem of recognizing the target speaker's speech in overlapping speech. Specifically, existing self-supervised learning (SSL) methods perform well when dealing with single-speaker speech but have limited ability to eliminate overlapping speech interference. This is particularly evident in the well-known "cocktail party problem," where multiple speakers talk simultaneously in a noisy environment. To tackle this challenge, the paper proposes a weakly supervised speech pre-training method—TS-HuBERT (Target-Speaker HuBERT), which improves the model's ability to recognize the target speaker in overlapping speech by introducing the target speaker's enrollment information as auxiliary input. ### Main Contributions: 1. **Weakly Supervised Speech Pre-training**: The paper explores a new direction of using additional weak label information (such as speaker identity) for speech pre-training. 2. **Target Speaker Enrollment Information**: By introducing the target speaker's enrollment audio as auxiliary input, the model is guided to learn the features of the target speaker. 3. **Improved HuBERT Model**: Based on the HuBERT model, the TS-HuBERT model is proposed, which learns to extract the target speaker's features from overlapping speech through a masked prediction task. 4. **Experimental Validation**: Experimental results on the Libri2Mix and WSJ0-2mix datasets show that the TS-HuBERT model significantly outperforms the existing SSL model WavLM in the target speaker recognition task. ### Method Overview: - **Model Design**: The TS-HuBERT model adopts a structure similar to HuBERT, including a Convolutional Neural Network (CNN) encoder and a Transformer encoder. During pre-training, the primary speech signal is mixed with interfering speech from different speakers, while providing the target speaker's enrollment audio as auxiliary input. - **Masked Prediction Loss**: The model learns to extract the target speaker's features from overlapping speech through a masked prediction task, calculating the cross-entropy loss only for the masked parts of the primary speech signal. - **Pre-training Data Preparation**: Overlapping speech data is generated through a dynamic mixing strategy, and the target speaker's enrollment audio is provided. - **Downstream Task Application**: The pre-trained model is applied to the target speaker recognition task through a linear projection layer or adaptation layer. ### Experimental Results: - On standard ASR tasks, the TS-HuBERT model performs comparably to existing SSL models. - On the target speaker recognition task, the TS-HuBERT model achieves new state-of-the-art performance (WER=6.1%) on the WSJ0-2mix dataset, significantly outperforming the WavLM model. - On the Libri2Mix dataset in noisy environments, the TS-HuBERT model also performs excellently, with a relative WER reduction of about 10%. In summary, the paper proposes an effective weakly supervised speech pre-training method that can better recognize the target speaker's speech in overlapping speech.

Weakly-Supervised Speech Pre-training: A Case Study on Target Speech Recognition

Unispeech-sat: Universal speech representation learning with speaker aware pre-training

Fast and Efficient Multilingual Self-Supervised Pre-training for Low-Resource Speech Recognition

Injecting Text in Self-Supervised Speech Pretraining

SA-WavLM: Speaker-Aware Self-Supervised Pre-training for Mixture Speech

A Joint Speech Enhancement and Self-Supervised Representation Learning Framework for Noise-Robust Speech Recognition

Feature Learning and Ensemble Pre-Tasks Based Self-Supervised Speech Denoising and Dereverberation

Wavlm: Large-scale self-supervised pre-training for full stack speech processing

Progressive Residual Extraction based Pre-training for Speech Representation Learning

A Noise-Robust Self-Supervised Pre-Training Model Based Speech Representation Learning for Automatic Speech Recognition.

Tts4pretrain 2.0: Advancing the Use of Text and Speech in ASR Pretraining with Consistency and Contrastive Losses

Improving Self-Supervised Learning for Speech Recognition with Intermediate Layer Supervision

Target Speech Extraction with Pre-trained Self-supervised Learning Models

Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition

W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training

Low-Resource Self-Supervised Learning with SSL-Enhanced TTS

Adapting Self-Supervised Models to Multi-Talker Speech Recognition Using Speaker Embeddings

Joint Training of Speech Enhancement and Self-supervised Model for Noise-robust ASR

Why Does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?