Abstract:Communication in the real world is inherently multimodal. When having a conversation, typically sighted and hearing people use both auditory and visual cues to understand one another. For example, objects may make sounds as they move in space, or we may use the movement of a person's mouth to better understand what they are saying in a noisy environment. Still, many neuroscience experiments rely on unimodal stimuli to understand encoding of sensory features in the brain. The extent to which visual information may influence encoding of auditory information and vice versa in natural environments is thus unclear. Here, we addressed this question by recording scalp electroencephalography (EEG) in 11 subjects as they listened to and watched movie trailers in audiovisual (AV), visual (V) only, and audio (A) only conditions. We then fit linear encoding models that described the relationship between the brain responses and the acoustic, phonetic, and visual information in the stimuli. We also compared whether auditory and visual feature tuning was the same when stimuli were presented in the original AV format versus when visual or auditory information was removed. In these stimuli, visual and auditory information was relatively uncorrelated, and included spoken narration over a scene as well as animated or live-action characters talking with and without their face visible. For this stimulus, we found that auditory feature tuning was similar in the AV and A-only conditions, and similarly, tuning for visual information was similar when stimuli were presented with the audio present (AV) and when the audio was removed (V only). In a cross prediction analysis, we investigated whether models trained on AV data predicted responses to A or V only test data similarly to models trained on unimodal data. Overall, prediction performance using AV training and V test sets was similar to using V training and V test sets, suggesting that the auditory information has a relatively smaller effect on EEG. In contrast, prediction performance using AV training and A only test set was slightly worse than using matching A only training and A only test sets. This suggests the visual information has a stronger influence on EEG, though this makes no qualitative difference in the derived feature tuning. In effect, our results show that researchers may benefit from the richness of multimodal datasets, which can then be used to answer more than one research question.

Trial-by-trial surprise-decoding model for visual and auditory binary oddball tasks

Decoding of auditory surprise in adult magnetoencephalography data using Bayesian models

Beyond Auditory Relay: Dissecting the Inferior Colliculus’s Role in Sensory Prediction, Cognitive Decision-Making, and Reward Prediction

Bayesian semantic surprise based on different types of regularities predicts the N400 and P600 brain signals

Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making

Stimulus history, not expectation, drives sensory prediction errors in mammalian cortex

Oddball evoked deviant responses reflect complex context dependent expectations in mouse V1

Calibrating Bayesian decoders of neural spiking activity

Auditory oddball responses in the human subthalamic nucleus and substantia nigra pars reticulata

Two distinct dynamic modes subtend the detection of unexpected sounds

Fast adaptation to rule switching using neuronal surprise

Single-trial decoding of auditory novelty responses facilitates the detection of residual consciousness

Revealing nonlinear neural decoding by analyzing choices

Bayesian Time-Series Classifier for Decoding Simple Visual Stimuli from Intracranial Neural Activity

Single trial Bayesian inference by population vector readout in the barn owl's sound localization system

Bayesian Neural Decoding Using A Diversity-Encouraging Latent Representation Learning Method

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts

Valence resolution of ambiguous facial expressions using an emotional oddball task.

Prediction across sensory modalities: A neurocomputational model of the McGurk effect

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features