Abstract:Abstract Perception of speech is improved when presentation of the audio signal is accompanied by concordant visual speech gesture information. This enhancement is most prevalent when the audio signal is degraded. One potential means by which the brain affords perceptual enhancement is thought to be through the integration of concordant information from multiple sensory channels in a common site of convergence, multisensory integration (MSI) sites. Some studies have identified potential sites in the superior temporal gyrus/sulcus (STG/S) that are responsive to multisensory information from the auditory speech signal and visual speech movement. One limitation of these studies is that they do not control for activity resulting from attentional modulation cued by such things as visual information signaling the onsets and offsets of the acoustic speech signal, as well as activity resulting from MSI of properties of the auditory speech signal with aspects of gross visual motion that are not specific to place of articulation information. This fMRI experiment uses spatial wavelet bandpass filtered Japanese sentences presented with background multispeaker audio noise to discern brain activity reflecting MSI induced by auditory and visual correspondence of place of articulation information that controls for activity resulting from the above-mentioned factors. The experiment consists of a low-frequency (LF) filtered condition containing gross visual motion of the lips, jaw, and head without specific place of articulation information, a midfrequency (MF) filtered condition containing place of articulation information, and an unfiltered (UF) condition. Sites of MSI selectively induced by auditory and visual correspondence of place of articulation information were determined by the presence of activity for both the MF and UF conditions relative to the LF condition. Based on these criteria, sites of MSI were found predominantly in the left middle temporal gyrus (MTG), and the left STG/S (including the auditory cortex). By controlling for additional factors that could also induce greater activity resulting from visual motion information, this study identifies potential MSI sites that we believe are involved with improved speech perception intelligibility.

Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information

Multisensory integration of speech and gestures in a naturalistic paradigm

Multisensory and modality specific processing of visual speech in different regions of the premotor cortex

Auditory, Visual and Audiovisual Speech Processing Streams in Superior Temporal Sulcus

Motor Circuit and Superior Temporal Sulcus Activities Linked to Individual Differences in Multisensory Speech Perception

Multisensory integration and motor resonance in the primary motor cortex

Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing

Dynamic Causal Modeling Analysis Reveals the Modulation of Motor Cortex and Integration in Superior Temporal Gyrus During Multisensory Speech Perception

Multisensory Integration of Naturalistic Speech and Gestures in Autistic Adults

A Double Dissociation between Anterior and Posterior Superior Temporal Gyrus for Processing Audiovisual Speech Demonstrated by Electrocorticography

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

Can you hear me yet? An intracranial investigation of speech and non-speech audiovisual interactions in human cortex

Retinotopic effects during spatial audio-visual integration

Quantified Acoustic–optical Speech Signal Incongruity Identifies Cortical Sites of Audiovisual Speech Processing

Auditory dyadic interactions through the “eye” of the social brain: How visual is the posterior STS interaction region?

The Contribution of Motion-Sensitive Brain Areas to Visual Speech Recognition

Multisensory integration of speech signals: the relationship between space and time

Prediction and constraint in audiovisual speech perception

The Causal Role of Left and Right Superior Temporal Gyri in Speech Perception in Noise: A Transcranial Magnetic Stimulation Study

Visual speech differentially modulates beta, theta, and high gamma bands in auditory cortex

Attention Drives Visual Processing and Audiovisual Integration During Multimodal Communication