Abstract:Target speaker extraction (TSE) which has the capability to directly extract desired speech given enrollment utterances of the target speaker has attracted more and more attention for its potential applications in solving the cocktail-party problem. Despite the considerable progress made by existing time-domain methods, which have become the dominant approach for TSE, these methods often significantly degrade their performance under more realistic conditions. This paper proposes an innovative approach in the time-frequency (T-F) domain, namely X-TF-GridNet, which uses complex spectrum mapping to extract the real and imaginary (RI) components of the target speech. Specifically, the TF-GridNet block was designed to serve as the primary speaker extractor module. Our proposed method boasts two key extensions: first, a U 2-Net style network adeptly extracts robust fixed speaker embeddings, which could efficiently capture and represent target speaker information. Second, an adaptive embedding fusion (AEA) mechanism ensures the effective utilization of target speaker information, which makes the backbone extractor focus on the speech of interest. Additionally, we also introduced a multi-task learning framework, comprising two distinct loss functions, to explicitly enhance both the discriminative speaker embeddings for the reference speech and the overall quality of the target speech. We conducted extensive ablation studies and quantitative comparisons against previous TSE methods on both the WSJ0-2mix and its noisy and reverberant counterparts. Our proposed method achieved a commendable SI-SDR of 19.7 dB with a moderate model size on the WSJ0-2mix dataset, and the SI-SDR can be improved to 20.7 dB with a larger model. Experimental results demonstrated that, compared with existing time-domain approaches, our proposed method not only achieved competitive performance across multiple objective metrics but also mitigated speaker confusion errors under more challenging conditions, including various interferences such as noises and reverberation.

Generation-Based Target Speech Extraction with Speech Discretization and Vocoder.

Continuous Target Speech Extraction: Enhancing Personalized Diarization and Extraction on Complex Recordings

Diffusion Conditional Expectation Model for Efficient and Robust Target Speech Extraction.

TSELM: Target Speaker Extraction using Discrete Tokens and Language Models

Improving Target Speaker Extraction with Sparse LDA-transformed Speaker Embeddings

X-Tf-Gridnet: A Time-Frequency Domain Target Speaker Extraction Network with Adaptive Speaker Embedding Fusion

3S-TSE: Efficient Three-Stage Target Speaker Extraction for Real-Time and Low-Resource Applications

WeSep: A Scalable and Flexible Toolkit Towards Generalizable Target Speaker Extraction

Libri2Vox Dataset: Target Speaker Extraction with Diverse Speaker Conditions and Synthetic Data

Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction

X-SepFormer: End-to-end Speaker Extraction Network with Explicit Optimization on Speaker Confusion

Exploring Time-Frequency Domain Target Speaker Extraction For Causal and Non-Causal Processing.

Audio-Visual Target Speaker Extraction with Reverse Selective Auditory Attention

DENSE: Dynamic Embedding Causal Target Speech Extraction

Speaker Extraction with Detection of Presence and Absence of Target Speakers

Cepstral Smoothing of Spectral Masks for Acoustic Vector-Sensor Based Convolutive Speech Separation

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

pTSE-T: Presentation Target Speaker Extraction using Unaligned Text Cues

Multi-Level Speaker Representation for Target Speaker Extraction

A Study of Multichannel Spatiotemporal Features and Knowledge Distillation on Robust Target Speaker Extraction

Target Sound Extraction with Variable Cross-modality Clues