Abstract:Conditional sound separation in multi-source audio mixtures without having access to single source sound data during training is a long standing challenge. Existing mix-and-separate based methods suffer from significant performance drop with multi-source training mixtures due to the lack of supervision signal for single source separation cases during training. However, in the case of language-conditional audio separation, we do have access to corresponding text descriptions for each audio mixture in our training data, which can be seen as (rough) representations of the audio samples in the language modality. To this end, in this paper, we propose a generic bi-modal separation framework which can enhance the existing unsupervised frameworks to separate single-source signals in a target modality (i.e., audio) using the easily separable corresponding signals in the conditioning modality (i.e., language), without having access to single-source samples in the target modality during training. We empirically show that this is well within reach if we have access to a pretrained joint embedding model between the two modalities (i.e., CLAP). Furthermore, we propose to incorporate our framework into two fundamental scenarios to enhance separation performance. First, we show that our proposed methodology significantly improves the performance of purely unsupervised baselines by reducing the distribution shift between training and test samples. In particular, we show that our framework can achieve 71% boost in terms of Signal-to-Distortion Ratio (SDR) over the baseline, reaching 97.5% of the supervised learning performance. Second, we show that we can further improve the performance of the supervised learning itself by 17% if we augment it by our proposed weakly-supervised framework, that enables a powerful semi-supervised framework for audio separation.

Semi-blind source separation via sparse representations and online dictionary learning

Blind source separation by sparse decomposition in a signal dictionary

Blind Source Separation with Compressively Sensed Based on SBL

Multi-Task Learning for Blind Source Separation

A smart universal single-channel blind source separation method and applications

Multichannel blind speech source separation with a disjoint constraint source model

Adaptive dictionary based approach for background noise and speaker classification and subsequent source separation

Blind nonnegative source separation using biological neural networks

Blind and Spatially-Regularized Online Joint Optimization of Source Separation, Dereverberation, and Noise Reduction

Blind Source Separation of Single-Channel Mixtures via Multi-Encoder Autoencoders

An online blind source separation for convolutive acoustic signals in frequency-domain

Data-Driven Source Separation Based on Simplex Analysis

Sparsity and adaptivity for the blind separation of partially correlated sources

Semi-Blind Source Separation with Learned Constraints

Blind Source Separation Based on Self-Organizing Neural Network.

Blind source separation and its development

Weakly-supervised Audio Separation via Bi-modal Semantic Similarity

Blind Separation Using Second Order Statistics for Non-Stationary Signals

Discussion about Development and Application of Blind Source Separation Technology

Determined Multichannel Blind Source Separation with Clustered Source Model

Quasi-Blind Source Separation Algorithm for Convolutive Mixture of Speech