Abstract:This paper describes sound event localization and detection (SELD) for spatial audio recordings captured by firstorder ambisonics (FOA) microphones. In this task, one may train a deep neural network (DNN) using FOA data annotated with the classes and directions of arrival (DOAs) of sound events. However, the performance of this approach is severely bounded by the amount of annotated data. To overcome this limitation, we propose a novel method of pretraining the feature extraction part of the DNN in a self-supervised manner. We use spatial audio-visual recordings abundantly available as virtual reality contents. Assuming that sound objects are concurrently observed by the FOA microphones and the omni-directional camera, we jointly train audio and visual encoders with contrastive learning such that the audio and visual embeddings of the same recording and DOA are made close. A key feature of our method is that the DOA-wise audio embeddings are jointly extracted from the raw audio data, while the DOA-wise visual embeddings are separately extracted from the local visual crops centered on the corresponding DOA. This encourages the latent features of the audio encoder to represent both the classes and DOAs of sound events. The experiment using the DCASE2022 Task 3 dataset of 20 hours shows non-annotated audio-visual recordings of 100 hours reduced the error score of SELD from 36.4 pts to 34.9 pts.

What problem does this paper attempt to address?

### Problems the paper attempts to solve The paper aims to solve the problem of insufficient data annotation in Sound Event Localization and Detection (SELD) tasks. Specifically, traditional SELD methods usually rely on a large amount of multi - channel audio data with annotations to train Deep Neural Networks (DNN), but obtaining such annotated data is very difficult and costly. To overcome this limitation, the authors propose a new self - supervised learning method, using rich Virtual Reality (VR) content for pre - training. ### Main contributions 1. **DOA - aware Audio - Visual Self - Supervised Learning (DOA - aware AV - SSL)** - By using spatial audio - visual recordings (e.g., VR content), this method jointly trains audio and visual encoders in the pre - training stage. - Through contrastive learning, the audio and visual embeddings of the same recording and Direction of Arrival (DOA) are made closer, while the embeddings of different recordings or DOAs are made farther apart. 2. **Audio embedding extraction on the DOA grid** - Unlike traditional methods, this method directly extracts audio embeddings on the DOA grid from the original FOA audio data, rather than through enhanced signals. - This method encourages the latent features of the audio encoder to represent both the category of sound events and the DOA simultaneously. 3. **Global contrastive learning** - To solve the problems of visible silent objects and occluded sound sources in local contrastive learning, global contrastive learning based on recordings is proposed. - Negative samples are generated by randomly rotating panoramic visual data, further encouraging the audio encoder to extract DOA information. ### Experimental results - Experiments were carried out using the DCASE2022 Task 3 dataset. The results show that by using 100 hours of unannotated audio - visual recordings, the error score of SELD is reduced from 36.4 to 34.9. - Compared with the traditional Audio - Visual Contrastive (AVC) method, the proposed DOA - aware method performs better in SELD tasks, especially in the case of multi - class and complex sound sources. ### Conclusion The method proposed in this paper effectively improves the performance of SELD tasks by using rich unannotated VR content for pre - training. Future work can further explore how to improve the performance of SELD in the case of insufficient annotated data, for example, by constructing spatial audio - visual datasets covering various fields to better match actual application scenarios.

DOA-Aware Audio-Visual Self-Supervised Learning for Sound Event Localization and Detection

Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection

Sound Event Localization based on Sound Intensity Vector Refined By DNN-Based Denoising and Source Separation

Zero- and Few-shot Sound Event Localization and Detection

STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events

Sound Event Localization and Detection Using Activity-Coupled Cartesian DOA Vector and RD3net

Two vs. Four-Channel Sound Event Localization and Detection

Joint Spatio-Temporal-Frequency Representation Learning for Improved Sound Event Localization and Detection

Sound Event Localization and Detection Based on Deep Learning

w2v-SELD: A Sound Event Localization and Detection Framework for Self-Supervised Spatial Audio Pre-Training

Enhanced Sound Event Localization and Detection in Real 360-degree audio-visual soundscapes

Sound Event Localization and Detection for Real Spatial Sound Scenes: Event-Independent Network and Data Augmentation Chains

Self-supervised Neural Audio-Visual Sound Source Localization via Probabilistic Spatial Modeling

Dynamic Kernel Convolution Network with Scene-dedicate Training for Sound Event Localization and Detection

Sound Event Detection and Localization with Distance Estimation

6DoF SELD: Sound Event Localization and Detection Using Microphones and Motion Tracking Sensors on self-motioning human

Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks

Sound Event Localization and Detection Based on Multiple DOA Beamforming and Multi-Task Learning

AD-YOLO: You Look Only Once in Training Multiple Sound Event Localization and Detection

HAAC: Hierarchical Audio Augmentation Chain for ACCDOA Described Sound Event Localization and Detection

ICASSP 2022 L3DAS22 Challenge: Ensemble of Resnet-Conformers with Ambisonics Data Augmentation for Sound Event Localization and Detection