Abstract:Abstract Perception of speech is improved when presentation of the audio signal is accompanied by concordant visual speech gesture information. This enhancement is most prevalent when the audio signal is degraded. One potential means by which the brain affords perceptual enhancement is thought to be through the integration of concordant information from multiple sensory channels in a common site of convergence, multisensory integration (MSI) sites. Some studies have identified potential sites in the superior temporal gyrus/sulcus (STG/S) that are responsive to multisensory information from the auditory speech signal and visual speech movement. One limitation of these studies is that they do not control for activity resulting from attentional modulation cued by such things as visual information signaling the onsets and offsets of the acoustic speech signal, as well as activity resulting from MSI of properties of the auditory speech signal with aspects of gross visual motion that are not specific to place of articulation information. This fMRI experiment uses spatial wavelet bandpass filtered Japanese sentences presented with background multispeaker audio noise to discern brain activity reflecting MSI induced by auditory and visual correspondence of place of articulation information that controls for activity resulting from the above-mentioned factors. The experiment consists of a low-frequency (LF) filtered condition containing gross visual motion of the lips, jaw, and head without specific place of articulation information, a midfrequency (MF) filtered condition containing place of articulation information, and an unfiltered (UF) condition. Sites of MSI selectively induced by auditory and visual correspondence of place of articulation information were determined by the presence of activity for both the MF and UF conditions relative to the LF condition. Based on these criteria, sites of MSI were found predominantly in the left middle temporal gyrus (MTG), and the left STG/S (including the auditory cortex). By controlling for additional factors that could also induce greater activity resulting from visual motion information, this study identifies potential MSI sites that we believe are involved with improved speech perception intelligibility.

The Contribution of Motion-Sensitive Brain Areas to Visual Speech Recognition

The contribution of visual areas to speech comprehension: a PET study in cochlear implants patients and normal-hearing subjects

Functional Connectivity between Face-Movement and Speech-Intelligibility Areas during Auditory-Only Speech Perception

Task-dependent modulation of the visual sensory thalamus assists visual-speech recognition

Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information

Prediction and constraint in audiovisual speech perception

Multisensory and modality specific processing of visual speech in different regions of the premotor cortex

Moving beyond the motor cortex: a brain-wide evaluation of target locations for intracranial speech neuroprostheses

Processing of Visual Speech Cues in Speech-in-Noise Comprehension Depends on Working Memory Capacity and Enhances Neural Speech Tracking in Older Adults With Hearing Impairment

Spoken language processing activates the primary visual cortex

Multisensory integration and motor resonance in the primary motor cortex

Phonetic versus spatial processes during motor‐oriented imitations of visuo‐labial and visuo‐lingual speech: A functional near‐infrared spectroscopy study

Visual speech differentially modulates beta, theta, and high gamma bands in auditory cortex

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

A Neuro-Experimental Evidence for the Motor Theory of Speech Perception

Utility of TMS to understand the neurobiology of speech

Auditory, Visual and Audiovisual Speech Processing Streams in Superior Temporal Sulcus

Primary Visual Cortex is Activated by Spoken Language Comprehension

A Comparison of Visual and Auditory Motion Processing in Human Cerebral Cortex

Processing communicative facial and vocal cues in the superior temporal sulcus

Multimodal comprehension in left hemisphere stroke patients