Abstract:Objective. To evaluate the potential of intracortical electrode array signals for brain-computer interfaces (BCIs) to restore lost speech, we measured the performance of decoders trained to discriminate a comprehensive basis set of 39 English phonemes and to synthesize speech sounds via a neural pattern matching method. We decoded neural correlates of spoken-out-loud words in the 'hand knob' area of precentral gyrus, a step toward the eventual goal of decoding attempted speech from ventral speech areas in patients who are unable to speak. Approach. Neural and audio data were recorded while two BrainGate2 pilot clinical trial participants, each with two chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English phonemes. Phoneme onsets were identified from audio recordings, and their identities were then classified from neural features consisting of each electrode's binned action potential counts or high-frequency local field potential power. Speech synthesis was performed using the 'Brain-to-Speech' pattern matching method. We also examined two potential confounds specific to decoding overt speech: acoustic contamination of neural signals and systematic differences in labeling different phonemes' onset times. Main results. A linear decoder achieved up to 29.3% classification accuracy (chance = 6%) across 39 phonemes, while an RNN classifier achieved 33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more electrodes or more training data, and that accuracy improved when utilizing time-varying structure in the data. Microphonic contamination and phoneme onset differences modestly increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a neural speech onset marker, respectively. Speech synthesis achieved r = 0.523 correlation between true and reconstructed audio. Significance. The ability to decode speech using intracortical electrode array signals from a nontraditional speech area suggests that placing electrode arrays in ventral speech areas is a promising direction for speech BCIs.

Decoding Articulation Motor Imagery Using Early Connectivity Information in the Motor Cortex: A Functional Near-Infrared Spectroscopy Study

Idle-state detection in motor imagery of articulation using early information: A functional Near-infrared spectroscopy study

Combining Motor Imagery with Selective Sensation Toward a Hybrid-Modality BCI

Decoding lexical tones and vowels in imagined tonal monosyllables using fNIRS signals

Motor Imagery Decoding Enhancement Based on Hybrid EEG-fNIRS Signals

The Effects of Different Brain Regions on fNIRS-based Task-state Detection in Speech Imagery

Application of a Common Spatial Pattern-Based Algorithm for an Fnirs-Based Motor Imagery Brain-Computer Interface.

Classification algorithm for motor imagery fusing CNN and attentional mechanisms based on functional near-infrared spectroscopy brain image

Imagined Speech Increases the Hemodynamic Response and Functional Connectivity of the Dorsal Motor Cortex.

Effect of functional and effective brain connectivity in identifying vowels from articulation imagery procedures

Motor Imagery BCI Classification Based on Multivariate Variational Mode Decomposition

Identification of Motor and Mental Imagery EEG in Two and Multiclass Subject-Dependent Tasks Using Successive Decomposition Index

Decoding of Voluntary and Involuntary Upper-Limb Motor Imagery Based on Graph Fourier Transform and Cross-Frequency Coupling Coefficients.

Multi-classification of Fnirs Signals in Four Body Parts Motor Imagery Tasks Measured from Motor Cortex

Integrating EEG and NIRS Improves BCI Performance During Motor Imagery

Imagined Speech and Visual Imagery as Intuitive Paradigms for Brain-Computer Interfaces

Extending Motor Imagery by Speech Imagery for Brain-Computer Interface.

Resting state EEG assisted imagined vowel phonemes recognition by native and non-native speakers using brain connectivity measures

Decoding spoken English phonemes from intracortical electrode arrays in dorsal precentral gyrus

Let ear speak:an auricular frontal-lobe-point's near-infrared spectroscope brain-computer interface

Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus