Abstract:Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time–frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time–frequency representation for speech separation, and the long latency in calculating the spectrograms. To address these shortcomings, we propose a fully convolutional time-domain audio separation network Conv-TasNet, a deep learning framework for end-to-end time-domain speech separation. Conv-TasNet uses a linear encoder to generate a representation of the speech waveform optimized for separating individual speakers. Speaker separation is achieved by applying a set of weighting functions masks to the encoder output. The modified encoder representations are then inverted back to the waveforms using a linear decoder. The masks are found using a temporal convolutional network consisting of stacked one-dimensional dilated convolutional blocks, which allows the network to model the long-term dependencies of the speech signal while maintaining a small model size. The proposed Conv-TasNet system significantly outperforms previous time–frequency masking methods in separating two- and three-speaker mixtures. Additionally, Conv-TasNet surpasses several ideal time–frequency magnitude masks in two-speaker speech separation as evaluated by both objective distortion measures and subjective quality assessment by human listeners. Finally, Conv-TasNet has a significantly smaller model size and a shorter minimum latency, making it a suitable solution for both offline and real-time speech separation applications. This study, therefore, represents a major step toward the realization of speech separation systems for real-world speech processing technologies.

A Distinct Synthesizer Convolutional Tasnet For Singing Voice Separation

Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation

Audiovisual Singing Voice Separation

Multi-Band Multi-Resolution Fully Convolutional Neural Networks for Singing Voice Separation

Comparison for Improvements of Singing Voice Detection System Based on Vocal Separation

Depthwise Separable Convolutions Versus Recurrent Neural Networks for Monaural Singing Voice Separation

Latent time-frequency component analysis: A novel pitch-based approach for singing voice separation

Research On Singing Voice Detection Based On A Long-Term Recurrent Convolutional Network With Vocal Separation And Temporal Smoothing

Gen-Res-Net: A Novel Generative Model for Singing Voice Separation

Spectral Mapping of Singing Voices: U-Net-Assisted Vocal Segmentation

A comprehensive study of speech separation: spectrogram vs waveform separation

Real-Time and Accurate: Zero-shot High-Fidelity Singing Voice Conversion with Multi-Condition Flow Synthesis

Towards Solving The Bottleneck Of Pitch-Based Singing Voice Separation

Neural Vocoder Feature Estimation for Dry Singing Voice Separation

MITAS: A Compressed Time-Domain Audio Separation Network with Parameter Sharing

Music Source Separation Based on a Lightweight Deep Learning Framework (DTTNET: DUAL-PATH TFC-TDF UNET)

Deep Learning Based Source Separation Applied To Choir Ensembles

Voice and accompaniment separation in music using self-attention convolutional neural network

Jointly Detecting and Separating Singing Voice: A Multi-Task Approach

Singer separation for karaoke content generation

Multi-Stage Non-Negative Matrix Factorization for Monaural Singing Voice Separation