Abstract:Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceived illusion-ba, and a reduced N1 when they perceived illusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENT The current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).

Those voices in your head: activation of auditory images during reading

Processing Speech and Thoughts during Silent Reading: Direct Reference Effects for Speech by Fictional Characters in Voice-Selective Auditory Cortex and a Theory-of-Mind Network

How silent is silent reading? Intracerebral evidence for top-down activation of temporal voice areas during reading

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Common and Distinct Neural Representations of Imagined and Perceived Speech

The role of visual imagery in story reading: Evidence from aphantasia

Auditory imagery ability influences accuracy when singing with altered auditory feedback

Mental Imagery Induces Cross-Modal Sensory Plasticity and Changes Future Auditory Perception

Speech imagery decoding as a window to speech planning and production

fMRI investigation of sentence comprehension by eye and by ear: Modality fingerprints on cognitive processes

Dissociable roles of neural pattern reactivation and transformation during recognition of words read aloud and silently: An MVPA study of the production effect

[The Neural Substrates for Voice Familiarity in Schizophrenic Patients with Auditory Verbal Hallucinations: an Event-Related Functional Magnetic Resonance Imaging Study].

Brain Activation Patterns in Normal Hearing Adults: An fNIRS Study Using an Adapted Clinical Speech Comprehension Task

Auditory perceptual assessment of voices: Examining perceptual ratings as a function of voice experience

Effect of functional and effective brain connectivity in identifying vowels from articulation imagery procedures

Language Experience Predicts Eye Movements During Online Auditory Comprehension

The Sounds of Sentences: Differentiating the Influence of Physical Sound, Sound Imagery, and Linguistically Implied Sounds on Physical Sound Processing

Imagined speech influences perceived loudness of sound

Predicting the subjective intensity of imagined experiences from electrophysiological measures of oscillatory brain activity

Task-dependent decoding of speaker and vowel identity from auditory cortical response patterns

Visual Imagery and False Memory for Pictures: A Functional Magnetic Resonance Imaging Study in Healthy Participants