Abstract:We present an end-to-end binaural audio rendering approach (Listen2Scene) for virtual reality (VR) and augmented reality (AR) applications. We propose a novel neural-network-based binaural sound propagation method to generate acoustic effects for indoor 3D models of real environments. Any clean audio or dry audio can be convolved with the generated acoustic effects to render audio corresponding to the real environment. We propose a graph neural network that uses both the material and the topology information of the 3D scenes and generates a scene latent vector. Moreover, we use a conditional generative adversarial network (CGAN) to generate acoustic effects from the scene latent vector. Our network can handle holes or other artifacts in the reconstructed 3D mesh model. We present an efficient cost function for the generator network to incorporate spatial audio effects. Given the source and the listener position, our learning-based binaural sound propagation approach can generate an acoustic effect in 0.1 milliseconds on an NVIDIA GeForce RTX 2080 Ti GPU. We have evaluated the accuracy of our approach with binaural acoustic effects generated using an interactive geometric sound propagation algorithm and captured real acoustic effects / real-world recordings. We also performed a perceptual evaluation and observed that the audio rendered by our approach is more plausible than audio rendered using prior learning-based and geometric-based sound propagation algorithms. We quantitatively evaluated the accuracy of our approach using statistical acoustic parameters, and energy decay curves. The demo videos, code and dataset are available online (<a class="link-external link-https" href="https://anton-jeran.github.io/Listen2Scene/" rel="external noopener nofollow">this https URL</a>).

SOAF: Scene Occlusion-aware Neural Acoustic Field

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis

Novel-View Acoustic Synthesis from 3D Reconstructed Rooms

AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Sound Adversarial Audio-Visual Navigation

SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound

A Scene Representation for Online Spatial Sonification

Unified Audio-Visual Saliency Model for Omnidirectional Videos with Spatial Audio

Room-aware portable Auditory Augmented Reality: Real-time spatial audio generation with geometric data analysis

Listen2Scene: Interactive material-aware binaural sound propagation for reconstructed 3D scenes

AudioEar: Single-View Ear Reconstruction for Personalized Spatial Audio

Neural Ambisonic Encoding For Multi-Speaker Scenarios Using A Circular Microphone Array

Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation

Audio-Visual Animation of Urban Space.

Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models

Hearing Anything Anywhere

Visar: Projecting Virtual Sound Spots for Acoustic Augmented Reality Using Air Nonlinearity

Towards Audio-Visual Saliency Prediction for Omnidirectional Video with Spatial Audio

Soundspaces 2.0: A simulation platform for visual-acoustic learning

Visual-guided scene-aware audio generation method based on hierarchical feature codec and rendering decision