Abstract:In this paper, we address the text-to-audio grounding issue, namely, grounding the segments of the sound event described by a natural language query in the untrimmed audio. This is a newly proposed but challenging audio-language task, since it requires to not only precisely localize all the on- and off-sets of the desired segments in the audio, but to perform comprehensive acoustic and linguistic understandings and reason the multimodal interactions between the audio and query. To tackle those problems, the existing method treats the query holistically as a single unit by a global query representation, which fails to highlight the keywords that contain rich semantics. Besides, this method has not fully exploited interactions between the query and audio. Moreover, since the audio and queries are arbitrary and variable in length, many meaningless parts of them are not filtered out in this method, which hinders the grounding of the desired segments. To this end, we propose a novel Query Graph with Cross-gating Attention (QGCA) model, which models the comprehensive relations between the words in query through a novel query graph. Besides, to capture the fine-grained interactions between audio and query, a cross-modal attention module that assigns higher weights to the keywords is introduced to generate the snippet-specific query representations. Finally, we also design a cross-gating module to emphasize the crucial parts as well as weaken the irrelevant ones in the audio and query. We extensively evaluate the proposed QGCA model on the public Audiogrounding dataset with significant improvements over several state-of-the-art methods. Moreover, further ablation study shows the consistent effectiveness of different modules in the proposed QGCA model.

GPA: Global and Prototype Alignment for Audio-Text Retrieval

AudioVSR: Enhancing Video Speech Recognition with Audio Data

Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrieval

Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition

Audio-text Retrieval with Transformer-based Hierarchical Alignment and Disentangled Cross-modal Representation

Improving Audio-Text Retrieval via Hierarchical Cross-Modal Interaction and Auxiliary Captions

Audio–text retrieval based on contrastive learning and collaborative attention mechanism

DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval

Bridging the Gap between Text, Audio, Image, and Any Sequence: A Novel Approach using Gloss-based Annotation

Multi-grained Representation Learning for Cross-modal Retrieval

Language-based Audio Retrieval with GPT-Augmented Captions and Self-Attended Audio Clips

Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training

STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment

AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head

Video-to-Audio Generation with Fine-grained Temporal Semantics

Mutual Alignment between Audiovisual Features for End-to-End Audiovisual Speech Recognition

Query-graph with Cross-gating Attention Model for Text-to-Audio Grounding.

Realization of Global Audio Telepresence Via a Learning-Based Model-Matching Approach with an Acoustic Array System

Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation

Improving Text-Audio Retrieval by Text-aware Attention Pooling and Prior Matrix Revised Loss

Audio-Text Retrieval in Context