Abstract:Sound propagation is the process by which sound energy travels through a medium, such as air, to the surrounding environment as sound waves. The room impulse response (RIR) describes this process and is influenced by the positions of the source and listener, the room's geometry, and its materials. Physics-based acoustic simulators have been used for decades to compute accurate RIRs for specific acoustic environments. However, we have encountered limitations with existing acoustic simulators. To address these limitations, we propose three novel solutions. First, we introduce a learning-based RIR generator that is two orders of magnitude faster than an interactive ray-tracing simulator. Our approach can be trained to input both statistical and traditional parameters directly, and it can generate both monaural and binaural RIRs for both reconstructed and synthetic 3D scenes. Our generated RIRs outperform interactive ray-tracing simulators in speech-processing applications, including ASR, Speech Enhancement, and Speech Separation. Secondly, we propose estimating RIRs from reverberant speech signals and visual cues without a 3D representation of the environment. By estimating RIRs from reverberant speech, we can augment training data to match test data, improving the word error rate of the ASR system. Our estimated RIRs achieve a 6.9% improvement over previous learning-based RIR estimators in far-field ASR tasks. We demonstrate that our audio-visual RIR estimator aids tasks like visual acoustic matching, novel-view acoustic synthesis, and voice dubbing, validated through perceptual evaluation. Finally, we introduce IR-GAN to augment accurate RIRs using real RIRs. IR-GAN parametrically controls acoustic parameters learned from real RIRs to generate new RIRs that imitate different acoustic environments, outperforming Ray-tracing simulators on the far-field ASR benchmark by 8.95%.

A Modified Radiosity Algorithm for Integrated Visual and Auditory Rendering

3D Audio Rendering in Distributed Virtual Environment

Hybrid Method of Image Source Model and Diffuse Surface Integration for the Simulation of Room Sound Field

Audio-Visual Animation of Urban Space.

Audio Rendering In Virtual Environment

Aural proxies and directionally-varying reverberation for interactive sound propagation in virtual environments

Hearing Anything Anywhere

Simplified model for generating 3D realistic sound in the multimedia and virtual reality systems

A New Radiosity Approach by Procedural Refinements for Realistic Image Sythesis

Three-Dimensional Acoustic Simulation Based on Virtual Environments.

VisibleSound: Perceiving Environmental Sound with 4D Form

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Acoustic Volume Rendering for Neural Impulse Response Fields

Computationally-efficient and perceptually-motivated rendering of diffuse reflections in room acoustics simulation

Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse

A real-time auralization framework for scientific experiments

Perceptual Content Authentication and Tamper Detection of Facial Image

Computer graphics visualization for acoustic simulation

Efficient learning-based sound propagation for virtual and real-world audio processing applications

Room-aware portable Auditory Augmented Reality: Real-time spatial audio generation with geometric data analysis

Sound Field Translation and Mixed Source Model for Virtual Applications with Perceptual Validation