Abstract:Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceived illusion-ba, and a reduced N1 when they perceived illusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENT The current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).

Filling-in of the Blindspot is Multisensory

Perceptual filling-in from the edge of the blind spot

Predictive Coding: A Possible Explanation of Filling-In at the Blind Spot

Simulation of Mono-eye Blind Spot Effect Based on Visual System

Stimulus-dependent delay of perceptual filling-in by microsaccades

Perceiving depth beyond sight: Evaluating intrinsic and learned cues via a proof of concept sensory substitution method in the visually impaired and sighted

Conscious vision in blindness: A new perceptual phenomenon implemented on the "wrong" side of the brain

The SSVEP tracks attention, not consciousness, during perceptual filling-in

Filling-in visual motion with sounds

Visual Mental Imagery and Neural Dynamics of Sensory Substitution in the Blindfolded Subject

Integration of auditory and visual cues in spatial navigation under normal and impaired viewing conditions

Sensitivity to visual features in inattentional blindness

Visual instrumental learning in blindsight monkeys

The Role of Spared Calcarine Cortex and Lateral Occipital Cortex in the Responses of Human Hemianopes to Visual Motion

What cortical areas are responsible for blindsight in hemianopic patients?

Filling gaps in visual motion for target capture

From blindsight to sight: cognitive rehabilitation of visual field defects

Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Network Localization of Unconscious Visual Perception in Blindsight

Functional specialization for auditory–spatial processing in the occipital cortex of congenitally blind humans