Abstract:Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceived illusion-ba, and a reduced N1 when they perceived illusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENT The current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).

Causality and cross-modal integration.

Visual determinants of a cross-modal illusion

Causality shifts the perceived temporal order of audiovisual events.

Auditory-visual integration during nonconscious perception

Locating the Sources for Cross-Modal Interactions and Decision Making During Judging the Visual-Affected Auditory Intensity Change.

What You See Depends on What You Hear: Temporal Averaging and Crossmodal Integration

Perceived multisensory common cause relations shape the ventriloquism effect but only marginally the trial-wise aftereffect

Congruency effects between auditory and tactile motion: extending the phenomenon of cross-modal dynamic capture

Modality-specific Sensory and Decisional Carryover Effects in Duration Perception

The Dynamics and Neural Correlates of Audio-Visual Integration Capacity as Determined by Temporal Unpredictability, Proactive Interference, and SOA

Multisensory integration of speech signals: the relationship between space and time

Splitting time: Sound-induced illusory visual temporal fission and fusion.

Grouping by feature of cross-modal flankers in temporal ventriloquism

Prior conscious experience modulates the impact of audiovisual temporal correspondence on unconscious visual processing

Cross-modal integration of auditory and visual motion signals

When eyes beat lips: speaker gaze affects audiovisual integration in the McGurk illusion

Intra- and Cross-Modal Cuing of Visual Spatial Attention: the Hyper-Effective Simultaneous Auditory Peripheral Cues

Sounds can boost the awareness of visual events through attention without cross-modal integration

Uncertainty-based causal inference modulates audiovisual temporal recalibration

Temporal ventriloquism along the path of apparent motion: speed perception under different spatial grouping principles

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding