Abstract:Communication in the real world is inherently multimodal. When having a conversation, typically sighted and hearing people use both auditory and visual cues to understand one another. For example, objects may make sounds as they move in space, or we may use the movement of a person's mouth to better understand what they are saying in a noisy environment. Still, many neuroscience experiments rely on unimodal stimuli to understand encoding of sensory features in the brain. The extent to which visual information may influence encoding of auditory information and vice versa in natural environments is thus unclear. Here, we addressed this question by recording scalp electroencephalography (EEG) in 11 subjects as they listened to and watched movie trailers in audiovisual (AV), visual (V) only, and audio (A) only conditions. We then fit linear encoding models that described the relationship between the brain responses and the acoustic, phonetic, and visual information in the stimuli. We also compared whether auditory and visual feature tuning was the same when stimuli were presented in the original AV format versus when visual or auditory information was removed. In these stimuli, visual and auditory information was relatively uncorrelated, and included spoken narration over a scene as well as animated or live-action characters talking with and without their face visible. For this stimulus, we found that auditory feature tuning was similar in the AV and A-only conditions, and similarly, tuning for visual information was similar when stimuli were presented with the audio present (AV) and when the audio was removed (V only). In a cross prediction analysis, we investigated whether models trained on AV data predicted responses to A or V only test data similarly to models trained on unimodal data. Overall, prediction performance using AV training and V test sets was similar to using V training and V test sets, suggesting that the auditory information has a relatively smaller effect on EEG. In contrast, prediction performance using AV training and A only test set was slightly worse than using matching A only training and A only test sets. This suggests the visual information has a stronger influence on EEG, though this makes no qualitative difference in the derived feature tuning. In effect, our results show that researchers may benefit from the richness of multimodal datasets, which can then be used to answer more than one research question.

A comparative study of eight human auditory models of monaural processing

Valid Acoustic Models of Cochlear Implants: One Size Does Not Fit All.

A framework for testing and comparing binaural models

Validation Of Acoustic Models Of Auditory Neural Prostheses

Towards a generalized monaural and binaural auditory model for psychoacoustics and speech intelligibility

One hundred ways to process time, frequency, rate and scale in the central auditory system: a pattern-recognition meta-analysis

Perceptual similarity between piano notes: Simulations with a template-based perception model

AMT 1.x: A toolbox for reproducible research in auditory modeling

A human ear simulation model for noise classification

Comparison of response properties of the electrically stimulated auditory nerve reported in human listeners and in animal models

On the value of diverse organisms in auditory research: From fish to flies to humans

A Biologically Plausible Spiking Model for Interaural Level Difference Processing Auditory Pathway in Human Brain

Calibration and reference simulations for the auditory periphery model of Verhulst et al. 2018 version 1.2

Model‐based signal processing for hearing aids

A Novel Approach to Investigate Subcortical and Cortical Sensitivity to Temporal Structure Simultaneously

A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts

Testing the dual-pathway model for auditory processing in human cortex

Comparative Study of State-based Neural Networks for Virtual Analog Audio Effects Modeling

Characteristic Analysis and Modeling for Signals of Auditory Propagation Pathway

The Human Auditory System and Audio

Computational model for synthesizing auditory brainstem responses to assess neuronal alterations in aging and autistic animal models