Abstract:Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceived illusion-ba, and a reduced N1 when they perceived illusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENT The current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).

Modulation of early auditory processing by visual information: Prediction or bimodal integration?

Asymmetrical Cross-Modal Influence on Neural Encoding of Auditory and Visual Features in Natural Scenes.

Retinotopic effects during spatial audio-visual integration

Prediction across sensory modalities: A neurocomputational model of the McGurk effect

Crossmodal visual predictions elicit spatially specific early visual cortex activity but later than real visual stimuli

Visual modulation of auditory evoked potentials in the cat

Effects of Temporal Congruity Between Auditory and Visual Stimuli Using Rapid Audio-Visual Serial Presentation

Electrophysiological Evidence of Crossmodal Correspondence Between Auditory Pitch and Visual Elevation Affecting Inhibition of Return.

Modification of Sudden Onset Auditory ERP by Involuntary Attention to Visual Stimuli.

The dissociation of semantically congruent and incongruent cross-modal effects on the visual attentional blink

The interplay between audiovisual temporal synchrony and semantic congruency in the cross-modal boost of the visual target discrimination during the attentional blink

Audiovisual Congruency and Incongruency Effects on Auditory Intensity Discrimination.

The Effect of Semantic Congruence for Visual-Auditory Bimodal Stimuli

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Inhibition of Return Decreases Early Audiovisual Integration: an Event-Related Potential Study.

The Time Course Of Spatial And Semantic Processing Between Audition And Vision: Evidence From Event-Related Potentials

Electrophysiological evidence of different neural processing between visual and audiovisual inhibition of return

Differential effects of intra-modal and cross-modal reward value on visual perception: ERP evidence

Quantified Acoustic–optical Speech Signal Incongruity Identifies Cortical Sites of Audiovisual Speech Processing

Audiovisual incongruence differentially impacts left and right hemisphere sensorimotor oscillations: Potential applications to production

Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study