Abstract:Binaural rendering is of great interest to virtual reality and immersive media. Although humans can naturally use their two ears to perceive the spatial information contained in sounds, it is a challenging task for machines to achieve binaural rendering since the description of a sound field often requires multiple channels and even the metadata of the sound sources. In addition, the perceived sound varies from person to person even in the same sound field. Previous methods generally rely on individual-dependent head-related transferred function (HRTF) datasets and optimization algorithms that act on HRTFs. In practical applications, there are two major drawbacks to existing methods. The first is a high personalization cost, as traditional methods achieve personalized needs by measuring HRTFs. The second is insufficient accuracy because the optimization goal of traditional methods is to retain another part of information that is more important in perception at the cost of discarding a part of the information. Therefore, it is desirable to develop novel techniques to achieve personalization and accuracy at a low cost. To this end, we focus on the binaural rendering of ambisonic and propose 1) channel-shared encoder and channel-compared attention integrated into neural networks and 2) a loss function quantifying interaural level differences to deal with spatial information. To verify the proposed method, we collect and release the first paired ambisonic-binaural dataset and introduce three metrics to evaluate the content information and spatial information accuracy of the end-to-end methods. Extensive experimental results on the collected dataset demonstrate the superior performance of the proposed method and the shortcomings of previous methods.

Adaptive subband partition encoding scheme for multiple audio objects using CNN and residual dense blocks mixture network

Low Bitrates Audio Object Coding Using Convolutional Auto-Encoder and Densenet Mixture Model.

Stacked Sparse Autoencoder for Audio Object Coding.

Sparse Autoencoder Based Multiple Audio Objects Coding Method

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

SpatialCodec: Neural Spatial Speech Coding

Scalable audio coding based on spatial perception in audio surveillance

Joint speech/audio coding based scalable perceptual audio coding

A Spatial Priority Based Scalable Audio Coding

A High Fidelity and Low Complexity Neural Audio Coding

A Bit Allocation Method for Object-Based Audio Coding Based on G.719 Codec

Frequency Domain Singular Value Decomposition for Efficient Spatial Audio Coding

Spatial grouping coding for three-dimensional multichannel audio system

A Lightweight and Real-Time Binaural Speech Enhancement Model with Spatial Cues Preservation

Neural Audio Coding with Deep Complex Networks

End-to-End Paired Ambisonic-Binaural Audio Rendering

Audio Tagging with Compact Feedforward Sequential Memory Network and Audio-to-Audio Ratio Based Data Augmentation

SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation

An Intra-BRNN and GB-RVQ Based END-TO-END Neural Audio Codec