Abstract:With the development of internet of things technologies, tremendous sensor audio data has been produced, which poses great challenges to audio-based event detection in smart cities. In this paper, we target a challenging audio-based event detection task, namely, text-to-audio grounding. In addition to precisely localizing all of the desired on- and off-sets in the untrimmed audio, this challenging new task requires extensive acoustic and linguistic comprehension as well as the reasoning for the crossmodal matching relations between the audio and query. The current approaches often treat the query as an entire one through a global query representation in order to address those issues. We contend that this strategy has several drawbacks. Firstly, the interactions between the query and the audio are not fully utilized. Secondly, it has not distinguished the importance of different keywords in a query. In addition, since the audio clips are of arbitrary lengths, there exist many segments which are irrelevant to the query but have not been filtered out in the approach. This further hinders the effective grounding of desired segments. Motivated by the above concerns, a novel Cross-modal Graph Interaction (CGI) model is proposed to comprehensively model the relations between the words in a query through a novel language graph. To capture the fine-grained relevances between the audio and query, a cross-modal attention module is introduced to generate snippet-specific query representations and automatically assign higher weights to keywords with more important semantics. Furthermore, we develop a cross-gating module for the audio and query to weaken irrelevant parts and emphasize the important ones.

Multi-level graph learning for audio event classification and human-perceived annoyance rating prediction

Joint Prediction of Audio Event and Annoyance Rating in an Urban Soundscape by Hierarchical Graph Representation Learning

Predicting Air Quality Using a Multi-Scale Spatiotemporal Graph Attention Network

Specialty may be better: A decoupling multi-modal fusion network for Audio-visual event localization

AI-based soundscape analysis: Jointly identifying sound sources and predicting annoyance

Multi-dimensional Edge-based Audio Event Relational Graph Representation Learning for Acoustic Scene Classification

Audio Event-Relational Graph Representation Learning for Acoustic Scene Classification

Effects of Soundscape Complexity on Urban Noise Annoyance Ratings: A Large-Scale Online Listening Experiment

Modeling Subjective Evaluation of Soundscape Quality in Urban Open Spaces: an Artificial Neural Network Approach

Listen As You Wish: Audio based Event Detection via Text-to-Audio Grounding in Smart Cities

Annoyance Prediction of Civil Aircraft Cabin Noise Based on Generalized Additive Models

Automated Audio Data Augmentation Network Using Bi-Level Optimization for Sound Event Localization and Detection

Attentional Graph Convolutional Network for Structure-aware Audio-Visual Scene Classification

Audio Related Quality of Experience Evaluation in Urban Transportation Environments with Brain Inspired Graph Learning

Interpretable Unsupervised Learning Framework for Multidimensional Erratic and Random Noise Attenuation

Sound quality metric indicators of rotorcraft noise annoyance using multilevel analysis

Artificial intelligence-based collaborative acoustic scene and event classification to support urban soundscape analysis and classification

Multi-label Feature selection with adaptive graph learning and label information enhancement

Asymptotic Consistent Graph Structure Learning for Multivariate Time-Series Anomaly Detection

Exploring Differences between Human Perception and Model Inference in Audio Event Recognition

Hierarchical graph contrastive learning of local and global presentation for multimodal sentiment analysis