Abstract:Deep learning has achieved a vital breakthrough in the sound source localization and has overcome the limitations of conventional model-based approaches. The deep learning-based method is to obtain a clear sound source distribution map from the information collected by a microphone array for identifying sound sources accurately. The performance of the deep learning-based method is closely related to the selections of input features and the architecture of networks. Consequently, a novel dynamic convolution-Transformer neural network (DYCTNN) is proposed to precisely estimate the number, positions, and strengths of multiple sound sources with high resolution. In this paper, the functional beamforming (FB) map is first served as input of networks. Then, a target function is tailored to generate the corresponding target map including the position and strength information of multiple sound sources as the ground truth for training the networks. Finally, the dynamic convolution-Transformer neural network with an encoder-decoder structure is constructed for the localization task of multiple sound sources. The proposed DYCTNN combines the advantages of the dynamic convolution and Transformer. The dynamic convolution can adaptively aggregate the multiple convolutional kernels according to the input maps to fully extract the local features, which effectively improves the representation capability of the network model for predicting the different spatial distribution characteristics of sound sources. Besides, the self-attention mechanism of the Transformer can effectively extract the global information of sound source spatial distribution from the FB map for improving the localization accuracy, and the depth-wise convolution can reduce the network model parameters and accelerate the training process. A numerical simulation with a 60-channel spiral microphone array is conducted to observe the localization capacity of the proposed DYCTNN. The quantitative and qualitative comparison results first prove that the proposed DYCTNN can predict the number, positions, and strengths of multiple sound sources very well, and its accuracy of sound source localization is better than that of the current deep learning-based method. In addition, the robustness and generalization of the proposed DYCTNN are also proven by comparing with the current deep learning-based method under several unseen acoustic conditions, such as signal–noise-ratio, measurement distance, frequency, and numbers of sound sources and microphones. Finally, the effectiveness of the dynamic convolution and self-attention mechanism in the proposed DYCTNN for multiple sound source localization is verified through an ablation study.

Localization, Detection and Tracking of Multiple Moving Sound Sources with a Convolutional Recurrent Neural Network

Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks

Attention mechanism combined with residual recurrent neural network for sound event detection and localization

Sound event localization and detection based on crnn using rectangular filters and channel rotation data augmentation

Spherical Convolutional Recurrent Neural Network for Real-Time Sound Source Tracking

Raw Waveform Based End-To-End Deep Convolutional Network For Spatial Localization Of Multiple Acoustic Sources

Robust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural Networks

Sound Event Detection Using Spatial Features and Convolutional Recurrent Neural Network

Permutation Invariant Recurrent Neural Networks for Sound Source Tracking Applications

SRP-DNN: Learning Direct-Path Phase Difference for Multiple Moving Sound Source Localization

Sound Source Distance Estimation in Diverse and Dynamic Acoustic Conditions

Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection

R-CRNN: Region-based Convolutional Recurrent Neural Network for Audio Event Detection

MTF-CRNN: Multiscale Time-Frequency Convolutional Recurrent Neural Network for Sound Event Detection.

A Study of Sound Source Localization Based on Improved Time Delay Features and the Convolutional Neural Network

Sound Event Localization and Detection Using Element-Wise Attention Gate and Asymmetric Convolutional Recurrent Neural Networks.

A Dynamic Convolution-Transformer Neural Network for Multiple Sound Source Localization Based on Functional Beamforming

Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments

Multi-Scale Convolutional Recurrent Neural Network with Ensemble Method for Weakly Labeled Sound Event Detection

U Recurrent Neural Network for Polyphonic Sound Event Detection and Localization

Exploiting Deep Neural Networks and Head Movements for Robust Binaural Localisation of Multiple Sources in Reverberant Environments