Abstract:Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceived illusion-ba, and a reduced N1 when they perceived illusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENT The current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).

Asymmetrical Cross-Modal Influence on Neural Encoding of Auditory and Visual Features in Natural Scenes.

Emergence of Neural Encoding of Auditory Objects While Listening to Competing Speakers

Crossmodal Integration Enhances Neural Representation of Task-Relevant Features in Audiovisual Face Perception

Neural Processing of Naturalistic Audiovisual Events in Space and Time

Audiovisual Emotional Congruency Modulates the Stimulus-Driven Cross-Modal Spread of Attention.

Crossmodal Modulation As a Basis for Visual Enhancement of Auditory Performance

Neural mechanisms of intermodal sustained selective attention with concurrently presented auditory and visual stimuli

Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement

A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts

Dissociable Neural Correlates of Multisensory Coherence and Selective Attention

The dissociation of semantically congruent and incongruent cross-modal effects on the visual attentional blink

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Congruent Audiovisual Speech Enhances Cortical Envelope Tracking During Auditory Selective Attention.

The effect of semantic content on the perception of audiovisual movieclips

Neural Correlates of Interobserver Visual Congruency in Free-Viewing Condition

Locating the Sources for Cross-Modal Interactions and Decision Making During Judging the Visual-Affected Auditory Intensity Change.

Changes of Effective Connectivity in the Alpha Band Characterize Differential Processing of Audiovisual Information in Cross-Modal Selective Attention

Attentional inhibition by alpha power is modulated by faster theta rhythm and audio-visual congruency during natural speech perception

N1 enhancement in synesthesia during visual and audio–visual perception in semantic cross-modal conflict situations: an ERP study

Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation.

What You See Depends on What You Hear: Temporal Averaging and Crossmodal Integration