Novel View Acoustic Parameter Estimation

Ricardo Falcon-Perez,Ruohan Gao,Gregor Mueckl,Sebastia V. Amengual Gari,Ishwarya Ananthabhotla

2024-10-31

Abstract:The task of Novel View Acoustic Synthesis (NVAS) - generating Room Impulse Responses (RIRs) for unseen source and receiver positions in a scene - has recently gained traction, especially given its relevance to Augmented Reality (AR) and Virtual Reality (VR) development. However, many of these efforts suffer from similar limitations: they infer RIRs in the time domain, which prove challenging to optimize; they focus on scenes with simple, single-room geometries; they infer only single-channel, directionally-independent acoustic characteristics; and they require inputs, such as 3D geometry meshes with material properties, that may be impractical to obtain for on-device applications. On the other hand, research suggests that sample-wise accuracy of RIRs is not required for perceptual plausibility in AR and VR. Standard acoustic parameters like Clarity Index (C50) or Reverberation Time (T60) have been shown to capably describe pertinent characteristics of the RIRs, especially late reverberation. To address these gaps, this paper introduces a new task centered on estimating spatially distributed acoustic parameters that can be then used to condition a simple reverberator for arbitrary source and receiver positions. The approach is modelled as an image-to-image translation task, which translates 2D floormaps of a scene into 2D heatmaps of acoustic parameters. We introduce a new, large-scale dataset of 1000 scenes consisting of complex, multi-room apartment conditions, and show that our method outperforms statistical baselines significantly. Moreover, we show that the method also works for directionally-dependent (i.e. beamformed) parameter prediction. Finally, the proposed method operates on very limited information, requiring only a broad outline of the scene and a single RIR at inference time.

Audio and Speech Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to estimate spatially - distributed acoustic parameters in new scenarios, especially to generate room impulse responses (RIRs) in augmented reality (AR) and virtual reality (VR) applications. Specifically, the paper proposes a new task - Novel - View Acoustic Parameter Estimation (NV - APE), which aims to predict 2D spatially - distributed acoustic parameters in a scene using limited geometric information (such as a simple floor plan) and a single reference RIR. This method aims to overcome the limitations of existing methods, such as insufficient ability to handle simple single - room geometries, only being able to infer single - channel, direction - independent acoustic properties, and requiring high - dimensional input data that is difficult to obtain (such as a complete 3D mesh containing material properties). The paper achieves its goal by modeling the task as an image - to - image conversion task, that is, converting a 2D floor plan into a 2D acoustic parameter heat map. In addition, the paper also introduces a large - scale dataset containing 1000 scenes under complex multi - room apartment conditions, and shows that the proposed method significantly outperforms other methods on the statistical baseline. Finally, the paper also shows that this method can be used for direction - dependent (i.e., beamforming) parameter prediction, and only requires minimal information input, namely the rough outline of the scene and a single RIR.

Novel View Acoustic Parameter Estimation

Novel-View Acoustic Synthesis from 3D Reconstructed Rooms

AV-RIR: Audio-Visual Room Impulse Response Estimation

Blind Acoustic Room Parameter Estimation Using Phase Features

Hearing Anything Anywhere

Blind Spatial Impulse Response Generation from Separate Room- and Scene-Specific Information

AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis

Few-Shot Audio-Visual Learning of Environment Acoustics

Realistic sources, receivers and walls improve the generalisability of virtually-supervised blind acoustic parameter estimators

Acoustic Volume Rendering for Neural Impulse Response Fields

Efficient learning-based sound propagation for virtual and real-world audio processing applications

Blind Room Parameter Estimation Using Multiple-Multichannel Speech Recordings

Deep Room Impulse Response Completion

Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects

Scene-Aware Audio Rendering via Deep Acoustic Analysis

NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields

3D Room Geometry Inference from Multichannel Room Impulse Response using Deep Neural Network

Blind Estimation of Sub-band Acoustic Parameters from Ambisonics Recordings using Spectro-Spatial Covariance Features

Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis

An end-to-end approach for blindly rendering a virtual sound source in an audio augmented reality environment