Abstract:Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceived illusion-ba, and a reduced N1 when they perceived illusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENT The current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).

Spatial alignment between faces and voices improves selective attention to audio-visual speech

Performance in an Audiovisual Selective Attention Task Using Speech-Like Stimuli Depends on the Talker Identities, But Not Temporal Coherence

Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading

Congruent Audiovisual Speech Enhances Cortical Envelope Tracking During Auditory Selective Attention.

The effects of temporal cues, point-light displays, and faces on speech identification and listening effort

Concurrent talking in immersive virtual reality: on the dominance of visual speech cues

Limits of Perceived Audio-Visual Spatial Coherence as Defined by Reaction Time Measurements

EXPRESS: Prior multisensory learning can facilitate auditory-only voice-identity and speech recognition in noise

Multisensory integration of speech signals: the relationship between space and time

Prediction and constraint in audiovisual speech perception

Attentional and linguistic interactions in speech perception

Dissociable Neural Correlates of Multisensory Coherence and Selective Attention

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Multisensory benefits for speech recognition in noisy environments

Attentional Modulation of the Early Cortical Representation of Speech Signals in Informational or Energetic Masking

Factors underlying masking release by voice-gender differences and spatial separation cues in multi-talker listening environments in listeners with and without hearing loss

Visual Facial Enhancements Can Significantly Improve Speech Perception in the Presence of Noise

Reassessing the Benefits of Audiovisual Integration to Speech Perception and Intelligibility

Attention Drives Visual Processing and Audiovisual Integration During Multimodal Communication

Individual Differences in Cognition and Perception Predict Neural Processing of Speech in Noise for Audiometrically Normal Listeners.

Congruent audiovisual speech enhances auditory attention decoding with EEG.