Abstract:In a naturalistic environment, auditory cues are often accompanied by information from other senses, which can be redundant with or complementary to the auditory information. Although the multisensory interactions derived from this combination of information and that shape auditory function are seen across all sensory modalities, our greatest body of knowledge to date centers on how vision influences audition. In this review, we attempt to capture the state of our understanding at this point in time regarding this topic. Following a general introduction, the review is divided into 5 sections. In the first section, we review the psychophysical evidence in humans regarding vision's influence in audition, making the distinction between vision's ability to enhance versus alter auditory performance and perception. Three examples are then described that serve to highlight vision's ability to modulate auditory processes: spatial ventriloquism, cross-modal dynamic capture, and the McGurk effect. The final part of this section discusses models that have been built based on available psychophysical data and that seek to provide greater mechanistic insights into how vision can impact audition. The second section reviews the extant neuroimaging and far-field imaging work on this topic, with a strong emphasis on the roles of feedforward and feedback processes, on imaging insights into the causal nature of audiovisual interactions, and on the limitations of current imaging-based approaches. These limitations point to a greater need for machine-learning-based decoding approaches toward understanding how auditory representations are shaped by vision. The third section reviews the wealth of neuroanatomical and neurophysiological data from animal models that highlights audiovisual interactions at the neuronal and circuit level in both subcortical and cortical structures. It also speaks to the functional significance of audiovisual interactions for two critically important facets of auditory perception-scene analysis and communication. The fourth section presents current evidence for alterations in audiovisual processes in three clinical conditions: autism, schizophrenia, and sensorineural hearing loss. These changes in audiovisual interactions are postulated to have cascading effects on higher-order domains of dysfunction in these conditions. The final section highlights ongoing work seeking to leverage our knowledge of audiovisual interactions to develop better remediation approaches to these sensory-based disorders, founded in concepts of perceptual plasticity in which vision has been shown to have the capacity to facilitate auditory learning.

What and where in the auditory systems of sighted and early blind individuals: Evidence from representational similarity analysis

How Does Experience Modulate Auditory Spatial Processing in Individuals with Blindness?

Testing the dual-pathway model for auditory processing in human cortex

The dual-pathway model of auditory signal processing

Prior Visual Experience Modulates Learning of Sound Localization among Blind Individuals.

Dual Representation of the Auditory Space

Early Cortical Connective Network Relating To Audiovisual Stimulation By Partial Directed Coherence Analysis

Cortical networks for audiovisual interactions during visual-affected auditory discrimination

Retinotopic effects during spatial audio-visual integration

The Way of the Light: How Visual Information Reaches the Auditory Cortex in Congenitally Deaf Adults

How Auditory Experience Differentially Influences the Function of Left and Right Superior Temporal Cortices

Functional Preference for Object Sounds and Voices in the Brain of Early Blind and Sighted Individuals

Functional specialization for auditory–spatial processing in the occipital cortex of congenitally blind humans

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

Auditory, Visual and Audiovisual Speech Processing Streams in Superior Temporal Sulcus

Cross-modal reorganization of horizontal connectivity in auditory cortex without altering thalamocortical projections

Cross-modal Representations in Early Visual and Auditory Cortices Revealed by Multi-Voxel Pattern Analysis.

The neural organization of visual information in the auditory cortex of the congenitally deaf

Language and Sensory Neural Plasticity in the Superior Temporal Cortex of the Deaf

Downward Cross-Modal Plasticity in Single-Sided Deafness

How Visual is the Visual Cortex? Comparing Connectional and Functional Fingerprints Between Congenitally Blind and Sighted Individuals