Abstract:Sound event localization and detection (SELD) has seen substantial advancements through learning-based methods. These systems, typically trained from scratch on specific datasets, have shown considerable generalization capabilities. Recently, deep neural networks trained on large-scale datasets have achieved remarkable success in the sound event classification (SEC) field, prompting an open question of whether these advancements can be extended to develop general-purpose SELD models. In this paper, leveraging the power of pre-trained SEC models, we propose pre-trained SELD networks (PSELDNets) on large-scale synthetic datasets. These synthetic datasets, generated by convolving sound events with simulated spatial room impulse responses (SRIRs), contain 1,167 hours of audio clips with an ontology of 170 sound classes. These PSELDNets are transferred to downstream SELD tasks. When we adapt PSELDNets to specific scenarios, particularly in low-resource data cases, we introduce a data-efficient fine-tuning method, AdapterBit. PSELDNets are evaluated on a synthetic-test-set using collected SRIRs from TAU Spatial Room Impulse Response Database (TAU-SRIR DB) and achieve satisfactory performance. We also conduct our experiments to validate the transferability of PSELDNets to three publicly available datasets and our own collected audio recordings. Results demonstrate that PSELDNets surpass state-of-the-art systems across all publicly available datasets. Given the need for direction-of-arrival estimation, SELD generally relies on sufficient multi-channel audio clips. However, incorporating the AdapterBit, PSELDNets show more efficient adaptability to various tasks using minimal multi-channel or even just monophonic audio clips, outperforming the traditional fine-tuning approaches.

Polyphonic sound event localization and detection using channel-wise FusionNet

A Polyphonic SELD Network Based on Attentive Feature Fusion and Multi-stage Training Strategy

Joint Spatio-Temporal-Frequency Representation Learning for Improved Sound Event Localization and Detection

Specialty may be better: A decoupling multi-modal fusion network for Audio-visual event localization

An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection

Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection

Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks

Sound Event Localization and Detection Based on Deep Learning

A Track-Wise Ensemble Event Independent Network for Polyphonic Sound Event Localization and Detection

Sound Event Localization and Detection Using Parallel Multi-attention Enhancement

PSELDNets: Pre-trained Neural Networks on Large-scale Synthetic Datasets for Sound Event Localization and Detection

MFF-EINV2: Multi-scale Feature Fusion across Spectral-Spatial-Temporal Domains for Sound Event Localization and Detection

Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy

Enhanced Sound Event Localization and Detection in Real 360-degree audio-visual soundscapes

Sound Event Localization and Detection Using a Spatial Omni-Dimensional Dynamic Interactions Network

Dynamic Kernel Convolution Network with Scene-dedicate Training for Sound Event Localization and Detection

A Model Ensemble Approach for Sound Event Localization and Detection.

Attention mechanism combined with residual recurrent neural network for sound event detection and localization

DOA-Aware Audio-Visual Self-Supervised Learning for Sound Event Localization and Detection

The NERCSLIP-USTC System for the L3DAS23 Challenge Task2: 3D Sound Event Localization and Detection (SELD)

Decoupling Temporal Convolutional Networks Model in Sound Event Detection and Localization