Abstract:Auditory assessments aim to quantify auditory abilities that impact real-world listening, such as replicating patient complaints of listening difficulty in specific situations or understanding how auditory systems perform in natural scenes. This goal suggests that assessments should be as realistic as possible. However, the need to quantify specific abilities requires greater experimental control than achievable in most natural settings. This fundamental trade-off between realism and control has existed since perception science began. But emerging technologies in virtual and augmented reality promise to bring the real world into the lab–or vice versa–and break this trade-off. Implemented correctly, these technologies offer new approaches to quasi-realistic auditory assessment with improved multisensory consistency, more natural tasks, and greater participant engagement that will make assessment more reliable, valid, and fun. Traditional laboratory-based testing remains the gold standard for auditory assessment in the clinic and in basic research. But several features of this approach limit the real-world validity of laboratory tasks. First, these tasks are rife with multisensory conflict. Booth walls and desktop displays of the testing environment are visually distracting and inconsistent with the auditory stimuli listeners are asked to judge, whereas real-world listening involves a consistent multisensory context that can reduce uncertainty and support task performance even when non-auditory cues are not explicitly informative about the task. Removing that support increases demands on participants' memory and mental imagery—cognitive skills that can confound attempts to measure sensory abilities. Even in explicitly multisensory studies, the spatial arrangement of auditory and visual stimuli are often poorly matched: Visuals may appear on a desktop display, while sounds appear inside the head or from a distant loudspeaker. In contrast, real-world auditory and visual features are aligned in all directions and remain so as a person moves around, producing a natural experience of immersion within the sensory world. Second, laboratory-based tasks are behaviorally unrealistic, typically presenting explicit feedback about an isolated task dimension such as localization or identification. But real-world listening combines multiple task dimensions and implicit feedback. Conversation in a crowded restaurant, for example, requires simultaneous localization, segregation, and identification of target speech, along with the evaluation of facial cues that might indicate which talker asked an important question but not the content of the question itself. Virtual reality (VR) and other immersive technologies have been gaining attention as new modes for computer-based entertainment, gaming, communication, and work. VR head-mounted displays (HMDs) present video and audio content (typically a simulated 3D environment) customized for the individual user. Low-latency tracking of the user's position in space is used to update the display in close to real time, allowing direct interaction with immersive multisensory experiences. At Boys Town National Research Hospital's Spatial Hearing Laboratory, we have been using commercial HMDs to explore the benefits of VR for auditory assessment. The focus has been on enhancing auditory assessment rather than on assessing the performance of virtual experiences (e.g., 3D audio) themselves. As such, our approach has been to add immersive multisensory elements without giving up the control made possible by existing unisensory methods and apparatus. For computer-based assessment, this is accomplished by software commands sent to and from a dedicated process that controls VR-based interactions with the participant. Across multiple investigations, we have noted several exciting potential benefits: Improving Multisensory Consistency. Replacing the sound booth with even a simple virtual scene can eliminate distracting visual elements and provide better consistency between visual and auditory elements, which can help participants understand the auditory stimulus even when visual cues are not directly informative. Working in my lab at Vanderbilt University, Travis Moore, AuD, PhD, conducted a binaural localization study in which listeners indicated the direction of a perceived sound by turning their heads in a simple immersive scene—a circular array of 360 balloons (one per degree azimuth). On each trial, they turned to face the balloon closest to where they heard a target sound and pressed a button to pop the selected balloon with a simple silent animation. This VR-based approach (a) provided a more consistent and controlled visual background than the sound booth interior, (b) supported participants' sense of target sounds' spatial position relative to their own dynamic viewpoint, and (c) provided naturalistic feedback on th -Abstract Truncated-

Comparison of a Head-Mounted Display and a Curved Screen in a Multi-Talker Audiovisual Listening Task

Comparison of 2D and 3D multichannel audio rendering methods for hearing research applications using technical and perceptual measures

Effect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments

Audiovisual angle and voice incongruence do not affect audiovisual verbal short-term memory in virtual reality

Validating a Method to Assess Lipreading, Audiovisual Gain, and Integration During Speech Reception With Cochlear-Implanted and Normal-Hearing Subjects Using a Talking Head

Using Virtual Reality to Assess Auditory Performance

Evaluating Spatial Hearing Using a Dual-Task Approach in a Virtual-Acoustics Environment

Active listening modulates the spatial hearing experience: a multicentric study

Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing

A Compact Two-Loudspeaker Virtual Sound Reproduction System for Clinical Testing of Spatial Hearing With Hearing-Assistive Devices

Combined assessment of auditory distance perception and externalization

Separate effects of auditory and visual room size on auditorium seat preference: a virtual reality study

Concurrent talking in immersive virtual reality: on the dominance of visual speech cues

Evaluation of spatial audio reproduction schemes for application in hearing aid research

Individualization Of Head-Related Transfer Function For Three-Dimensional Virtual Auditory Display: A Review

Comparison of subjective evaluations in virtual and real environments for soundscape researcha)

The effect of distance on audiovisual temporal integration in an outdoor virtual environment

Evaluation of Virtual Acoustic Environments with Different Acoustic Level of Detail

Performance on clinical outcomes, activities of daily living and user experience on head‐mounted displays for people with vision impairment

Congruent audiovisual speech enhances auditory attention decoding with EEG.

Do Congruent Auditory Stimuli Facilitate Visual Search in Dynamic Environments? an Experimental Study Based on Multisensory Interaction.