Abstract:Objective: This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training. Approach: We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes, by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train both subject-specific models using data from a single participant as well as multi-patient models exploiting data from multiple participants. Main Results: The subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi- subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation. Significance: The proposed SwinTW decoder enables future speech neuropros- theses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests the exciting possibility of developing speech neuropros- theses for people with speech disability without relying on their own neural data for training, which is not always feasible.

Swin fMRI Transformer Predicts Early Neurodevelopmental Outcomes from Neonatal fMRI

SwiFT: Swin 4D fMRI Transformer

Joint Prediction of Longitudinal Development of Cortical Surfaces and White Matter Fibers from Neonatal MRI

Transformer based multi-modal MRI fusion for prediction of post-menstrual age and neonatal brain development analysis

Swin Transformer for Fast MRI

Classification and Diagnosis of Autism Spectrum Disorder Using Swin Transformer.

Brain Structural Connectivity Guided Vision Transformers for Identification of Functional Connectivity Characteristics in Preterm Neonates

A novel Swin transformer approach utilizing residual multi-layer perceptron for diagnosing brain tumors in MRI images

Enhanced Infant Movement Analysis Using Transformer-Based Fusion of Diverse Video Features For Neurodevelopmental Monitoring

A Hybrid Multishape Learning Framework for Longitudinal Prediction of Cortical Surfaces and Fiber Tracts Using Neonatal Data

Transformer3: A Pure Transformer Framework for fMRI-Based Representations of Human Brain Function

Early Prediction of Cognitive Deficit in Very Preterm Infants Using Brain Structural Connectome with Transfer Learning Enhanced Deep Convolutional Neural Networks.

Generative AI for Rapid Diffusion MRI with Improved Image Quality, Reliability and Generalizability

Learning Infant Brain Developmental Connectivity for Cognitive Score Prediction.

A Multi-Task, Multi-Stage Deep Transfer Learning Model for Early Prediction of Neurodevelopment in Very Preterm Infants

Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals

Swin Transformer Improves the IDH Mutation Status Prediction of Gliomas Free of MRI-Based Tumor Segmentation

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

DFC-Igloo: A Dynamic Functional Connectome Learning Framework for Identifying Neurodevelopmental Biomarkers in Very Preterm Infants

SwinDTI: swin transformer-based generalized fast estimation of diffusion tensor parameters from sparse data

Community-Aware Transformer for Autism Prediction in fMRI Connectome