Abstract:In a naturalistic environment, auditory cues are often accompanied by information from other senses, which can be redundant with or complementary to the auditory information. Although the multisensory interactions derived from this combination of information and that shape auditory function are seen across all sensory modalities, our greatest body of knowledge to date centers on how vision influences audition. In this review, we attempt to capture the state of our understanding at this point in time regarding this topic. Following a general introduction, the review is divided into 5 sections. In the first section, we review the psychophysical evidence in humans regarding vision's influence in audition, making the distinction between vision's ability to enhance versus alter auditory performance and perception. Three examples are then described that serve to highlight vision's ability to modulate auditory processes: spatial ventriloquism, cross-modal dynamic capture, and the McGurk effect. The final part of this section discusses models that have been built based on available psychophysical data and that seek to provide greater mechanistic insights into how vision can impact audition. The second section reviews the extant neuroimaging and far-field imaging work on this topic, with a strong emphasis on the roles of feedforward and feedback processes, on imaging insights into the causal nature of audiovisual interactions, and on the limitations of current imaging-based approaches. These limitations point to a greater need for machine-learning-based decoding approaches toward understanding how auditory representations are shaped by vision. The third section reviews the wealth of neuroanatomical and neurophysiological data from animal models that highlights audiovisual interactions at the neuronal and circuit level in both subcortical and cortical structures. It also speaks to the functional significance of audiovisual interactions for two critically important facets of auditory perception-scene analysis and communication. The fourth section presents current evidence for alterations in audiovisual processes in three clinical conditions: autism, schizophrenia, and sensorineural hearing loss. These changes in audiovisual interactions are postulated to have cascading effects on higher-order domains of dysfunction in these conditions. The final section highlights ongoing work seeking to leverage our knowledge of audiovisual interactions to develop better remediation approaches to these sensory-based disorders, founded in concepts of perceptual plasticity in which vision has been shown to have the capacity to facilitate auditory learning.

Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex

Spatiotemporal models for multisensory integration

Neural Integration of Audiovisual Sensory Inputs in Macaque Amygdala and Adjacent Regions

The auditory-visual multisensory neurons and auditory-visual information integration in rat cortex

Modulation of auditory responses by visual inputs in the mouse auditory cortex

Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation.

Multi-timescale neural dynamics for multisensory integration

Recent advances in understanding the auditory cortex

Flexible integration of natural stimuli by auditory cortical neurons

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

Behavioral Indices of Multisensory Integration: Orientation to Visual Cues is Affected by Auditory Stimuli

Closing the loop on multisensory interactions: A neural architecture for multisensory causal inference and recalibration

Spectral and Temporal Integration in the Auditory Cortex of Alert Primates

Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception

Nonlinear Spectrotemporal Interactions Underlying Selectivity for Complex Sounds in Auditory Cortex

Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study

Neural correlates of multisensory integration in the human brain: an ALE meta-analysis

Morphology and Dendrite-Specific Synaptic Properties of Midbrain Neurons Shape Multimodal Integration

Stability of cross-sensory input to primary somatosensory cortex across experience

Compensating Level-Dependent Frequency Representation in Auditory Cortex by Synaptic Integration of Corticocortical Input

Nonlinear visuoauditory integration in the mouse superior colliculus