Abstract:Binaural rendering is of great interest to virtual reality and immersive media. Although humans can naturally use their two ears to perceive the spatial information contained in sounds, it is a challenging task for machines to achieve binaural rendering since the description of a sound field often requires multiple channels and even the metadata of the sound sources. In addition, the perceived sound varies from person to person even in the same sound field. Previous methods generally rely on individual-dependent head-related transferred function (HRTF) datasets and optimization algorithms that act on HRTFs. In practical applications, there are two major drawbacks to existing methods. The first is a high personalization cost, as traditional methods achieve personalized needs by measuring HRTFs. The second is insufficient accuracy because the optimization goal of traditional methods is to retain another part of information that is more important in perception at the cost of discarding a part of the information. Therefore, it is desirable to develop novel techniques to achieve personalization and accuracy at a low cost. To this end, we focus on the binaural rendering of ambisonic and propose 1) channel-shared encoder and channel-compared attention integrated into neural networks and 2) a loss function quantifying interaural level differences to deal with spatial information. To verify the proposed method, we collect and release the first paired ambisonic-binaural dataset and introduce three metrics to evaluate the content information and spatial information accuracy of the end-to-end methods. Extensive experimental results on the collected dataset demonstrate the superior performance of the proposed method and the shortcomings of previous methods.

Binaural Speech Enhancement Based On Dnn For The Application Of Virtual Reality

Binaural Reverberant Speech Separation Based on Deep Neural Networks

Binaural Deep Neural Network for Robust Speech Enhancement

An RNN-based Speech Enhancement Method for a Binaural Hearing Aid System

Binaural Classification for Reverberant Speech Segregation Using Deep Neural Networks

Binaural Rendering of Ambisonic Signals by Neural Networks

Deep Learning Based Binaural Speech Separation in Reverberant Environments

Neural Ambisonic Encoding For Multi-Speaker Scenarios Using A Circular Microphone Array

A Supervised Speech Enhancement Method for Smartphone-Based Binaural Hearing Aids

A Lightweight and Real-Time Binaural Speech Enhancement Model with Spatial Cues Preservation

End-to-End Paired Ambisonic-Binaural Audio Rendering

Deep Neural Network Based Noised Asian Speech Enhancement and Its Implementation on a Hearing Aid App.

SE Territory: Monaural Speech Enhancement Meets the Fixed Virtual Perceptual Space Mapping

Binaural Speech Enhancement Using Deep Complex Convolutional Transformer Networks

Deep Encoder/decoder Dual-Path Neural Network for Speech Separation in Noisy Reverberation Environments

Real-time multichannel deep speech enhancement in hearing aids: Comparing monaural and binaural processing in complex acoustic scenarios

A Unified DNN Approach to Speaker-Dependent Simultaneous Speech Enhancement and Speech Separation in Low SNR Environments

Research on Speech Separation Technology Based on Deep Learning

A DNN Parameter Mask for the Binaural Reverberant Speech Segregation

Speech separation based on reliable binaural cues with two-stage neural network in noisy-reverberant environments

A Single-Input/Binaural-Output Antiphasic Speech Enhancement Method for Speech Intelligibility Improvement.