Abstract:Objective. To evaluate the potential of intracortical electrode array signals for brain-computer interfaces (BCIs) to restore lost speech, we measured the performance of decoders trained to discriminate a comprehensive basis set of 39 English phonemes and to synthesize speech sounds via a neural pattern matching method. We decoded neural correlates of spoken-out-loud words in the 'hand knob' area of precentral gyrus, a step toward the eventual goal of decoding attempted speech from ventral speech areas in patients who are unable to speak. Approach. Neural and audio data were recorded while two BrainGate2 pilot clinical trial participants, each with two chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English phonemes. Phoneme onsets were identified from audio recordings, and their identities were then classified from neural features consisting of each electrode's binned action potential counts or high-frequency local field potential power. Speech synthesis was performed using the 'Brain-to-Speech' pattern matching method. We also examined two potential confounds specific to decoding overt speech: acoustic contamination of neural signals and systematic differences in labeling different phonemes' onset times. Main results. A linear decoder achieved up to 29.3% classification accuracy (chance = 6%) across 39 phonemes, while an RNN classifier achieved 33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more electrodes or more training data, and that accuracy improved when utilizing time-varying structure in the data. Microphonic contamination and phoneme onset differences modestly increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a neural speech onset marker, respectively. Speech synthesis achieved r = 0.523 correlation between true and reconstructed audio. Significance. The ability to decode speech using intracortical electrode array signals from a nontraditional speech area suggests that placing electrode arrays in ventral speech areas is a promising direction for speech BCIs.

Spoken Sentences Decoding Based on Intracranial High Gamma Response Using Dynamic Time Warping

Low-frequency Cortical Activity Reflects Context-dependent Parsing of Word Sequences

Speech neuromuscular decoding based on spectrogram images using conformal predictors with Bi-LSTM.

Identification of Attended Speech Stream Using Single-Trial Electroencephalography Recording

High Gamma Oscillations Enhance the Subdural Visual Speller

Decoding of Chinese Phoneme Clusters Using ECoG

Revealing Spatiotemporal Brain Dynamics Of Speech Production Based On Eeg And Eye Movement

Speech decoding using cortical and subcortical electrophysiological signals

Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus

Deciphering Phonemes from Syllables in Blood Oxygenation Level-Dependent Signals in Human Superior Temporal Gyrus

Intracranial Mapping of Response Latencies and Task Effects for Spoken Syllable Processing in the Human Brain

Task-dependent decoding of speaker and vowel identity from auditory cortical response patterns

Speech Activity Detection from Stereotactic EEG

Decoding spoken English phonemes from intracortical electrode arrays in dorsal precentral gyrus

Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity

Decoding Single and Paired Phonemes Using 7T Functional MRI

Continuous and discrete decoding of overt speech with electroencephalography

A Double Dissociation between Anterior and Posterior Superior Temporal Gyrus for Processing Audiovisual Speech Demonstrated by Electrocorticography

Neural Speech Decoding During Audition, Imagination and Production

Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

Continuous and discrete decoding of overt speech with scalp electroencephalography (EEG)