Abstract:Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceived illusion-ba, and a reduced N1 when they perceived illusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENT The current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).

Sensitivity of N400 Effect During Speech Comprehension under the Uni- and Bi-Modality Conditions

Working Memory Asymmetrically Modulates Auditory and Linguistic Processing of Speech.

Correlation Between Audio–visual Enhancement of Speech in Different Noise Environments and SNR: A Combined Behavioral and Electrophysiological Study

EEG Gamma-Band Activity During Audiovisual Speech Comprehension in Different Noise Environments.

Interaction between Phonological and Semantic Processes in Visual Word Recognition using Electrophysiology

Audiovisual perception of interrupted speech by nonnative listeners

Neural semantic effects of tone accents

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Auditory Dominance in Processing Chinese Semantic Abnormalities in Response to Competing Audio-visual Stimuli.

The N400 and Post-N400 positivity effect in Mandarin classifier-noun congruence: An ERP study

Effects of degraded speech processing and binaural unmasking investigated using functional near-infrared spectroscopy (fNIRS)

Parafoveal and foveal N400 effects in natural reading: A timeline of semantic processing from fixation‐related potentials

Cognitive integration of asynchronous natural or non-natural auditory and visual information in videos of real-world events: an event-related potential study

Neural Correlates of Audiovisual Integration of Semantic Category Information.

Time-varying networks of ERPs in P300-speller paradigms based on spatially and semantically congruent audiovisual bimodality

Selective Audiovisual Semantic Integration Enabled by Feature-Selective Attention.

Individual Differences in Cognition and Perception Predict Neural Processing of Speech in Noise for Audiometrically Normal Listeners.

Interactions Between Modal and Amodal Semantic Areas in Spoken Word Comprehension.

Processing of Visual Speech Cues in Speech-in-Noise Comprehension Depends on Working Memory Capacity and Enhances Neural Speech Tracking in Older Adults With Hearing Impairment

Speaker-listener neural coupling reveals a right-lateralized mechanism for non-native speech-in-noise comprehension

Speaker-Listener Neural Coupling Reveals an Adaptive Mechanism for Speech Comprehension in a Noisy Environment