Abstract:Abstract Perception of speech is improved when presentation of the audio signal is accompanied by concordant visual speech gesture information. This enhancement is most prevalent when the audio signal is degraded. One potential means by which the brain affords perceptual enhancement is thought to be through the integration of concordant information from multiple sensory channels in a common site of convergence, multisensory integration (MSI) sites. Some studies have identified potential sites in the superior temporal gyrus/sulcus (STG/S) that are responsive to multisensory information from the auditory speech signal and visual speech movement. One limitation of these studies is that they do not control for activity resulting from attentional modulation cued by such things as visual information signaling the onsets and offsets of the acoustic speech signal, as well as activity resulting from MSI of properties of the auditory speech signal with aspects of gross visual motion that are not specific to place of articulation information. This fMRI experiment uses spatial wavelet bandpass filtered Japanese sentences presented with background multispeaker audio noise to discern brain activity reflecting MSI induced by auditory and visual correspondence of place of articulation information that controls for activity resulting from the above-mentioned factors. The experiment consists of a low-frequency (LF) filtered condition containing gross visual motion of the lips, jaw, and head without specific place of articulation information, a midfrequency (MF) filtered condition containing place of articulation information, and an unfiltered (UF) condition. Sites of MSI selectively induced by auditory and visual correspondence of place of articulation information were determined by the presence of activity for both the MF and UF conditions relative to the LF condition. Based on these criteria, sites of MSI were found predominantly in the left middle temporal gyrus (MTG), and the left STG/S (including the auditory cortex). By controlling for additional factors that could also induce greater activity resulting from visual motion information, this study identifies potential MSI sites that we believe are involved with improved speech perception intelligibility.

Auditory, Visual and Audiovisual Speech Processing Streams in Superior Temporal Sulcus

Processing communicative facial and vocal cues in the superior temporal sulcus

A Double Dissociation between Anterior and Posterior Superior Temporal Gyrus for Processing Audiovisual Speech Demonstrated by Electrocorticography

Convergence of spoken and written language processing in the superior temporal sulcus

Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing

Mouth and voice: A relationship between visual and auditory preference in the human superior temporal sulcus

Auditory dyadic interactions through the “eye” of the social brain: How visual is the posterior STS interaction region?

fMRI-guided TMS of the superior temporal sulcus impairs multisensory temporal processing

Rostro-caudal organization of the human posterior superior temporal sulcus revealed by connectivity profiles.

Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information

Multiple Cortical Visual Streams in Humans

A multimodal interface for speech perception: the role of the left superior temporal sulcus in social cognition and autism

Converging Evidence From Electrocorticography and BOLD fMRI for a Sharp Functional Boundary in Superior Temporal Gyrus Related to Multisensory Speech Processing

Phonological representations of auditory and visual speech in the occipito-temporal cortex and beyond

Cortical Dynamics of Acoustic and Phonological Processing in Speech Perception

What and where in the auditory systems of sighted and early blind individuals: Evidence from representational similarity analysis

Controlling for individual differences in fMRI brain activation to tones, syllables, and words

Multisensory and modality specific processing of visual speech in different regions of the premotor cortex

Dynamic Causal Modeling Analysis Reveals the Modulation of Motor Cortex and Integration in Superior Temporal Gyrus During Multisensory Speech Perception

The Causal Role of Left and Right Superior Temporal Gyri in Speech Perception in Noise: A Transcranial Magnetic Stimulation Study

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution