Abstract:This paper proposes to use similarities of audio captions for estimating audio-caption relevances to be used for training text-based audio retrieval systems. Current audio-caption datasets (e.g., Clotho) contain audio samples paired with annotated captions, but lack relevance information about audio samples and captions beyond the annotated ones. Besides, mainstream approaches (e.g., CLAP) usually treat the annotated pairs as positives and consider all other audio-caption combinations as negatives, assuming a binary relevance between audio samples and captions. To infer the relevance between audio samples and arbitrary captions, we propose a method that computes non-binary audio-caption relevance scores based on the textual similarities of audio captions. We measure textual similarities of audio captions by calculating the cosine similarity of their Sentence-BERT embeddings and then transform these similarities into audio-caption relevance scores using a logistic function, thereby linking audio samples through their annotated captions to all other captions in the dataset. To integrate the computed relevances into training, we employ a listwise ranking objective, where relevance scores are converted into probabilities of ranking audio samples for a given textual query. We show the effectiveness of the proposed method by demonstrating improvements in text-based audio retrieval compared to methods that use binary audio-caption relevances for training.

Audio–text retrieval based on contrastive learning and collaborative attention mechanism

Language-based Audio Retrieval with Co-Attention Networks

Cacophony: An Improved Contrastive Audio-Text Model

Contrastive Latent Space Reconstruction Learning for Audio-Text Retrieval

Audio-Text Retrieval in Context

Multi-grained Representation Learning for Cross-modal Retrieval

Speaker-Text Retrieval via Contrastive Learning

Retrieval-Augmented Text-to-Audio Generation

Audio-Text Models Do Not Yet Leverage Natural Language

Text-based Audio Retrieval by Learning from Similarities between Audio Captions

Improving Audio-Text Retrieval via Hierarchical Cross-Modal Interaction and Auxiliary Captions

Enhancing Audio Retrieval with Attention-based Encoder for Audio Feature Representation

Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrieval

On Metric Learning for Audio-Text Cross-Modal Retrieval

Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

Voice Keyword Retrieval Method Using Attention Mechanism and Multimodal Information Fusion

Bridging Language Gaps in Audio-Text Retrieval

Dissecting Temporal Understanding in Text-to-Audio Retrieval

Introducing Auxiliary Text Query-modifier to Content-based Audio Retrieval

Unsupervised Improvement of Audio-Text Cross-Modal Representations

Language-based Audio Retrieval with GPT-Augmented Captions and Self-Attended Audio Clips