Abstract:Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceived illusion-ba, and a reduced N1 when they perceived illusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENT The current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).

Imagined speech influences perceived loudness of sound

Emergence of Neural Encoding of Auditory Objects While Listening to Competing Speakers

Robust cortical encoding of slow temporal modulations of speech.

Common and Distinct Neural Representations of Imagined and Perceived Speech

Working Memory Asymmetrically Modulates Auditory and Linguistic Processing of Speech.

The Sounds of Sentences: Differentiating the Influence of Physical Sound, Sound Imagery, and Linguistically Implied Sounds on Physical Sound Processing

Neural representations of imagined speech revealed by frequency-tagged magnetoencephalography responses

Mental Imagery Induces Cross-Modal Sensory Plasticity and Changes Future Auditory Perception

Attention is Required for Knowledge-Based Sequential Grouping: Insights from the Integration of Syllables into Words.

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Neurophysiological evidence for goal-oriented modulation of speech perception

Cerebral bases of subliminal speech priming

Attentional and linguistic interactions in speech perception

Imagined speech can be decoded from low- and cross-frequency intracranial EEG features

Processing Speech and Thoughts during Silent Reading: Direct Reference Effects for Speech by Fictional Characters in Voice-Selective Auditory Cortex and a Theory-of-Mind Network

Auditory-motor entrainment and listening experience shape the perceptual learning of concurrent speech

Sensory and Perceptual Decisional Processes Underlying the Perception of Reverberant Auditory Environments

Selective auditory attention within naturalistic scenes modulates reactivity to speech sounds

Subjective signal strength distinguishes reality from imagination

Neural interference between real and imagined visual stimuli

The Same Ultra-Rapid Parallel Brain Dynamics Underpin the Production and Perception of Speech