Abstract:Target speech separation refers to extracting the target speaker's speech from mixed signals. Despite the recent advances in deep learning based close-talk speech separation, the applications to real-world are still an open issue. Two main challenges are the complex acoustic environment and the real-time processing requirement. To address these challenges, we propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture in reverberant environments, assisted with directional information of the speaker(s). Firstly, against variations brought by complex environment, the key idea is to increase the acoustic representation completeness through the jointly modeling of temporal, spectral and spatial discriminability between the target and interference source. Specifically, temporal, spectral, spatial along with the designed directional features are integrated to create a joint acoustic representation. Secondly, to reduce the latency, we design a fully-convolutional autoencoder framework, which is purely end-to-end and single-pass. All the feature computation is implemented by the network layers and operations to speed up the separation procedure. Evaluation is conducted on simulated reverberant dataset WSJ0-2mix and WSJ0-3mix under speaker-independent scenario. Experimental results demonstrate that the proposed method outperforms state-of-the-art deep learning based multi-channel approaches with fewer parameters and faster processing speed. Furthermore, the proposed temporal-spatial neural filter can handle mixtures with varying and unknown number of speakers and exhibits persistent performance even when existing a direction estimation error. Codes and models will be released soon.

Conformer Space Neural Architecture Search for Multi-Task Audio Separation

On Time Domain Conformer Models for Monaural Speech Separation in Noisy Reverberant Acoustic Environments

Hierarchic Temporal Convolutional Network With Cross-Domain Encoder for Music Source Separation

Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation

Neural Network Alternatives to Convolutive Audio Models for Source Separation

WA-Transformer: Window Attention-based Transformer with Two-stage Strategy for Multi-task Audio Source Separation

Improved Conformer-based End-to-End Speech Recognition Using Neural Architecture Search

Efficient Gradient-Based Neural Architecture Search for End-to-End ASR

Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search for End-to-End ASR

Separate-to-Recognize: Joint Multi-target Speech Separation and Speech Recognition for Speaker-attributed ASR

Speaker-conditioning Single-channel Target Speaker Extraction using Conformer-based Architectures

Source-Aware Context Network for Single-Channel Multi-Speaker Speech Separation.

3M: Multi-loss, Multi-path and Multi-level Neural Networks for speech recognition

An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits

Listening and Grouping: an Online Autoregressive Approach for Monaural Speech Separation

CONTINUOUS SPEECH SEPARATION WITH CONFORMER

A New Neural Beamformer for Multi-channel Speech Separation

Multi-Dimensional and Multi-Scale Modeling for Speech Separation Optimized by Discriminative Learning

SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation

Towards A Unified Conformer Structure: from ASR to ASV Task

Multi-stage music separation network with dual-branch attention and hybrid convolution