Sound source localization based on multi-task learning and image translation network

Yifan Wu,Roshan Ayyalasomayajula,Michael J Bianco,Dinesh Bharadia,Peter Gerstoft

DOI: https://doi.org/10.1121/10.0007133

Abstract:Supervised learning-based sound source localization (SSL) methods have been shown to achieve a promising localization accuracy in the past. In this paper, MTIT, SSL for indoors using Multi-Task learning and Image Translation network, an image translation-based deep neural networks (DNNs) framework for SSL is presented to predict the locations of sound sources with random positions in a continuous space. We extract and represent the spatial features of the sound signals as beam response at each direction which can indicate the chance of the source in each point of the room. We utilize the multi-task learning (MTL) based training framework. There are one encoder and two decoders in our DNN. The encoder aims to obtain a compressed representation of the input beamspectrum surfaces while the two decoders focus on two tasks in parallel. One decoder focuses on resolving the multipath caused by reverberation and the other decoder predicts the source location. Since these two decoders share the same encoder, by training these two decoders in parallel, the shared representations are refined. We comprehensively evaluate the localization performance of our method in the simulated data, measured impulse response and real recordings datasets and compare it with multiple signal classification, steered response power with phase transform, and a competing convolutional neural network approach. It turns out that MTIT can outperform all of the baseline methods in a dynamic environment and also can achieve a good generalization performance.