Abstract:Conditional sound separation in multi-source audio mixtures without having access to single source sound data during training is a long standing challenge. Existing mix-and-separate based methods suffer from significant performance drop with multi-source training mixtures due to the lack of supervision signal for single source separation cases during training. However, in the case of language-conditional audio separation, we do have access to corresponding text descriptions for each audio mixture in our training data, which can be seen as (rough) representations of the audio samples in the language modality. To this end, in this paper, we propose a generic bi-modal separation framework which can enhance the existing unsupervised frameworks to separate single-source signals in a target modality (i.e., audio) using the easily separable corresponding signals in the conditioning modality (i.e., language), without having access to single-source samples in the target modality during training. We empirically show that this is well within reach if we have access to a pretrained joint embedding model between the two modalities (i.e., CLAP). Furthermore, we propose to incorporate our framework into two fundamental scenarios to enhance separation performance. First, we show that our proposed methodology significantly improves the performance of purely unsupervised baselines by reducing the distribution shift between training and test samples. In particular, we show that our framework can achieve 71% boost in terms of Signal-to-Distortion Ratio (SDR) over the baseline, reaching 97.5% of the supervised learning performance. Second, we show that we can further improve the performance of the supervised learning itself by 17% if we augment it by our proposed weakly-supervised framework, that enables a powerful semi-supervised framework for audio separation.

CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction

Language-Queried Target Sound Extraction Without Parallel Training Data

Target Sound Extraction with Variable Cross-modality Clues

T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining

CLAP: Contrastive Language-Audio Pre-training Model for Multi-modal Sentiment Analysis.

CLAP: Learning Audio Concepts From Natural Language Supervision

Learning Speech Representation From Contrastive Token-Acoustic Pretraining

tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models

SeisCLIP: A Seismology Foundation Model Pre-Trained by Multimodal Data for Multipurpose Seismic Feature Extraction

DSCLAP: Domain-Specific Contrastive Language-Audio Pre-Training

Masked Audio Modeling with CLAP and Multi-Objective Learning

Separate Anything You Describe

CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training

Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training

Prior-agnostic Multi-scale Contrastive Text-Audio Pre-training for Parallelized TTS Frontend Modeling

ParaCLAP -- Towards a general language-audio model for computational paralinguistic tasks

Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

Target Speaker Extraction with Curriculum Learning

Improving curriculum learning for target speaker extraction with synthetic speakers

Weakly-supervised Audio Separation via Bi-modal Semantic Similarity

Audio-free Prompt Tuning for Language-Audio Models