Abstract:Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time–frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time–frequency representation for speech separation, and the long latency in calculating the spectrograms. To address these shortcomings, we propose a fully convolutional time-domain audio separation network Conv-TasNet, a deep learning framework for end-to-end time-domain speech separation. Conv-TasNet uses a linear encoder to generate a representation of the speech waveform optimized for separating individual speakers. Speaker separation is achieved by applying a set of weighting functions masks to the encoder output. The modified encoder representations are then inverted back to the waveforms using a linear decoder. The masks are found using a temporal convolutional network consisting of stacked one-dimensional dilated convolutional blocks, which allows the network to model the long-term dependencies of the speech signal while maintaining a small model size. The proposed Conv-TasNet system significantly outperforms previous time–frequency masking methods in separating two- and three-speaker mixtures. Additionally, Conv-TasNet surpasses several ideal time–frequency magnitude masks in two-speaker speech separation as evaluated by both objective distortion measures and subjective quality assessment by human listeners. Finally, Conv-TasNet has a significantly smaller model size and a shorter minimum latency, making it a suitable solution for both offline and real-time speech separation applications. This study, therefore, represents a major step toward the realization of speech separation systems for real-world speech processing technologies.

Probabilistic Binary-Mask Cocktail-Party Source Separation in a Convolutional Deep Neural Network

Deep Transform: Cocktail Party Source Separation via Probabilistic Re-Synthesis

Deep Karaoke: Extracting Vocals from Musical Mixtures Using a Convolutional Deep Neural Network

Joint Deep Neural Network for Single-Channel Speech Separation on Masking-Based Training Targets

Cracking the cocktail party problem by multi-beam deep attractor network

End-to-end Networks for Supervised Single-channel Speech Separation

Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation

Speaker and Direction Inferred Dual-channel Speech Separation

Towards Solving Cocktail-Party: The First Method to Build a Realistic Dataset with Ground Truths for Speech Separation

On the Use of Deep Mask Estimation Module for Neural Source Separation Systems.

Improving Source Separation via Multi-Speaker Representations

Blind Source Separation Using Mixtures of Alpha-Stable Distributions

Real-time Speech Enhancement and Separation with a Unified Deep Neural Network for Single/Dual Talker Scenarios

Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation

A Speaker-Dependent Approach to Single-Channel Joint Speech Separation and Acoustic Modeling Based on Deep Neural Networks for Robust Recognition of Multi-Talker Speech

Single-Microphone Speech Enhancement and Separation Using Deep Learning

Speech separation of a target speaker based on deep neural networks

Speech Separation Based on Signal-Noise-dependent Deep Neural Networks for Robust Speech Recognition

Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation

Convolutional Maxout Neural Networks for Speech Separation

Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition