Abstract:Experimentalists studying multisensory integration compare neural responses to multisensory stimuli with responses to the component modalities presented in isolation. This procedure is problematic for multisensory speech perception since audiovisual speech and auditory-only speech are easily intelligible but visual-only speech is not. To overcome this confound, we developed intracranial encephalography (iEEG) deconvolution. Individual stimuli always contained both auditory and visual speech, but jittering the onset asynchrony between modalities allowed for the time course of the unisensory responses and the interaction between them to be independently estimated. We applied this procedure to electrodes implanted in human epilepsy patients (both male and female) over the posterior superior temporal gyrus (pSTG), a brain area known to be important for speech perception. iEEG deconvolution revealed sustained positive responses to visual-only speech and larger, phasic responses to auditory-only speech. Confirming results from scalp EEG, responses to audiovisual speech were weaker than responses to auditory-only speech, demonstrating a subadditive multisensory neural computation. Leveraging the spatial resolution of iEEG, we extended these results to show that subadditivity is most pronounced in more posterior aspects of the pSTG. Across electrodes, subadditivity correlated with visual responsiveness, supporting a model in which visual speech enhances the efficiency of auditory speech processing in pSTG. The ability to separate neural processes may make iEEG deconvolution useful for studying a variety of complex cognitive and perceptual tasks. SIGNIFICANCE STATEMENT Understanding speech is one of the most important human abilities. Speech perception uses information from both the auditory and visual modalities. It has been difficult to study neural responses to visual speech because visual-only speech is difficult or impossible to comprehend, unlike auditory-only and audiovisual speech. We used intracranial encephalography deconvolution to overcome this obstacle. We found that visual speech evokes a positive response in the human posterior superior temporal gyrus, enhancing the efficiency of auditory speech processing.

Speech imagery decoding as a window to speech planning and production

Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

Imagined speech can be decoded from low- and cross-frequency intracranial EEG features

Common and Distinct Neural Representations of Imagined and Perceived Speech

Parallel or sequential? Decoding conceptual and phonological/phonetic information from MEG signals during language production

Imagined speech event detection from electrocorticography and its transfer between speech modes and subjects

What do you have in mind? ERP markers of visual and auditory imagery

Neural representations of imagined speech revealed by frequency-tagged magnetoencephalography responses

Imagined Speech and Visual Imagery as Intuitive Paradigms for Brain-Computer Interfaces

Revealing Spatiotemporal Brain Dynamics Of Speech Production Based On Eeg And Eye Movement

Neural Speech Decoding During Audition, Imagination and Production

Decoding speech perception from non-invasive brain recordings

[Key technology of brain-computer interaction based on speech imagery]

Online internal speech decoding from single neurons in a human participant

Individual Word Classification During Imagined Speech Using Intracranial Recordings

Speech decoding using cortical and subcortical electrophysiological signals

Electroencephalogram (EEG) Based Imagined Speech Decoding and Recognition

Decoding Articulation Motor Imagery Using Early Connectivity Information in the Motor Cortex: A Functional Near-Infrared Spectroscopy Study

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text

Decoding Single and Paired Phonemes Using 7T Functional MRI