Abstract:We propose a novel data-driven approach to single-channel speech separation based on deep neural networks (DNNs) to directly model the highly nonlinear relationship between speech features of a mixed signal containing a target speaker and other interfering speakers. We focus our discussion on a semisupervised mode to separate speech of the target speaker from an unknown interfering speaker, which is more flexible than the conventional supervised mode with known information of both the target and interfering speakers. Two key issues are investigated. First, we propose a DNN architecture with dual outputs of the features of both the target and interfering speakers, which is shown to achieve a better generalization capability than that with output features of only the target speaker. Second, we propose using a set of multiple DNNs, each intending to be signal-noise-dependent (SND), to cope with the difficulty that one single general DNN could not well accommodate all the speaker mixing variabilities at different signal-to-noise ratio (SNR) levels. Experimental results on the speech separation challenge (SSC) data demonstrate that our proposed framework achieves better separation results than other conventional approaches in a supervised or semisupervised mode. SND-DNNs could also yield significant performance improvements over a general DNN for speech separation in low SNR cases. Furthermore, for automatic speech recognition (ASR) following speech separation, this purely front-end processing with a single set of speaker-independent ASR acoustic models, achieves a relative word error rate (WER) reduction of 11.6% over a state-of-the-art separation and recognition system where a complicated joint back-end decoding framework with multiple sets of speaker-dependent ASR acoustic models needs to be implemented. When speaker-adaptive ASR acoustic models for the target speakers are adopted for the enhanced signals, another 12.1% WER reduction over our best speaker-independent ASR system is achieved.

Neural Spatial Filter: Target Speaker Speech Separation Assisted with Directional Information

Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation

3D Spatial Features for Multi-Channel Target Speech Separation

Robust Spatial Filtering Network for Separating Speech in the Direction of Interest

Multi-channel Speech Separation Using Spatially Selective Deep Non-linear Filters

SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation

Enhancing End-to-End Multi-channel Speech Separation Via Spatial Feature Learning

Boosting Spatial Information for Deep Learning Based Multichannel Speaker-Independent Speech Separation in Reverberant Environments.

Speaker and Direction Inferred Dual-channel Speech Separation

Localization Based Stereo Speech Source Separation Using Probabilistic Time-Frequency Masking and Deep Neural Networks

Spatially Selective Deep Non-linear Filters for Speaker Extraction

Speech separation of a target speaker based on deep neural networks

A Target Speaker Separation Neural Network with Joint-Training

Enhanced Neural Beamformer with Spatial Information for Target Speech Extraction

Localization Based Stereo Speech Separation Using Deep Networks.

Locate and Beamform: Two-dimensional Locating All-neural Beamformer for Multi-channel Speech Separation

Source-Aware Context Network for Single-Channel Multi-Speaker Speech Separation.

Real-time binaural speech separation with preserved spatial cues

A Regression Approach to Single-Channel Speech Separation Via High-Resolution Deep Neural Networks.

Multi-Target Ensemble Learning for Monaural Speech Separation

Binaural Reverberant Speech Separation Based on Deep Neural Networks