Abstract:Objective: This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training. Approach: We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes, by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train both subject-specific models using data from a single participant as well as multi-patient models exploiting data from multiple participants. Main Results: The subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi- subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation. Significance: The proposed SwinTW decoder enables future speech neuropros- theses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests the exciting possibility of developing speech neuropros- theses for people with speech disability without relying on their own neural data for training, which is not always feasible.

Generalized neural decoders for transfer learning across participants and recording modalities

Neural Decoding Based on Probabilistic Neural Network

Neural decoding from stereotactic EEG: accounting for electrode variability across subjects

Deep learning approaches for neural decoding across architectures and recording modalities

Efficient Neural Decoding Based on Multimodal Training

Deep Learning for Micro-Electrocorticographic (µecog) Data

Group-level Brain Decoding with Deep Learning

Toward Generalizing Visual Brain Decoding to Unseen Subjects

Evaluating the structure of cognitive tasks with transfer learning

Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals

Exploiting correlations across trials and behavioral sessions to improve neural decoding

Meeting brain–computer interface user performance expectations using a deep neural network decoding framework

Making brain-machine interfaces robust to future neural variability

A novel deep transfer learning framework integrating general and domain-specific features for EEG-based brain–computer interface

Cross-paradigm pretraining of convolutional networks improves intracranial EEG decoding

A Novel Multiscale Dilated Convolution Neural Network with Gating Mechanism for Decoding Driving Intentions Based on EEG

Decoding speech from non-invasive brain recordings

Unsupervised decoding of long-term, naturalistic human neural recordings with automated video and audio annotations

A neural speech decoding framework leveraging deep learning and speech synthesis

Continuous Tracking using Deep Learning-based Decoding for Non-invasive Brain-Computer Interface

Transfer Learning-Based Behavioural Task Decoding from Brain Activity