Abstract:Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time–frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time–frequency representation for speech separation, and the long latency in calculating the spectrograms. To address these shortcomings, we propose a fully convolutional time-domain audio separation network Conv-TasNet, a deep learning framework for end-to-end time-domain speech separation. Conv-TasNet uses a linear encoder to generate a representation of the speech waveform optimized for separating individual speakers. Speaker separation is achieved by applying a set of weighting functions masks to the encoder output. The modified encoder representations are then inverted back to the waveforms using a linear decoder. The masks are found using a temporal convolutional network consisting of stacked one-dimensional dilated convolutional blocks, which allows the network to model the long-term dependencies of the speech signal while maintaining a small model size. The proposed Conv-TasNet system significantly outperforms previous time–frequency masking methods in separating two- and three-speaker mixtures. Additionally, Conv-TasNet surpasses several ideal time–frequency magnitude masks in two-speaker speech separation as evaluated by both objective distortion measures and subjective quality assessment by human listeners. Finally, Conv-TasNet has a significantly smaller model size and a shorter minimum latency, making it a suitable solution for both offline and real-time speech separation applications. This study, therefore, represents a major step toward the realization of speech separation systems for real-world speech processing technologies.

Time-Domain Speech Separation Networks With Graph Encoding Auxiliary

Time Domain Speech Enhancement Using Self-Attention-Based Subspace Projection

Time-Domain Mapping with Convolution Networks for End-to-End Monaural Speech Separation

Multi-layer encoder-decoder time-domain single channel speech separation

Efficient time-domain speech separation using short encoded sequence network

Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation

Improved Speech Separation with Time-and-Frequency Cross-domain Joint Embedding and Clustering

ForkNet: Simultaneous Time and Time-Frequency Domain Modeling for Speech Enhancement

An End-to-End Speech Separation Method Based on Features of Two Domains

Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation

Stepwise-Refining Speech Separation Network via Fine-Grained Encoding in High-Order Latent Domain

SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation

End-to-end Networks for Supervised Single-channel Speech Separation

Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation

Efficient Monaural Speech Separation with Multiscale Time-Delay Sampling

On End-to-end Multi-channel Time Domain Speech Separation in Reverberant Environments

Efficient Encoder-Decoder and Dual-Path Conformer for Comprehensive Feature Learning in Speech Enhancement

Multi-Dimensional and Multi-Scale Modeling for Speech Separation Optimized by Discriminative Learning

On Data Sampling Strategies for Training Neural Network Speech Separation Models

FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks

Multi-Scale Group Transformer for Long Sequence Modeling in Speech Separation