Abstract:We propose using subject embeddings to learn and exploit the structure in between‐subject variability in MEG decoding. We show that the combination of deep learning and subject embedding is crucial to closing the performance gap between subject‐ and group‐level models, with group models outperforming subject models on low‐accuracy subjects. Decoding brain imaging data are gaining popularity, with applications in brain‐computer interfaces and the study of neural representations. Decoding is typically subject‐specific and does not generalise well over subjects, due to high amounts of between subject variability. Techniques that overcome this will not only provide richer neuroscientific insights but also make it possible for group‐level models to outperform subject‐specific models. Here, we propose a method that uses subject embedding, analogous to word embedding in natural language processing, to learn and exploit the structure in between‐subject variability as part of a decoding model, our adaptation of the WaveNet architecture for classification. We apply this to magnetoencephalography data, where 15 subjects viewed 118 different images, with 30 examples per image; to classify images using the entire 1 s window following image presentation. We show that the combination of deep learning and subject embedding is crucial to closing the performance gap between subject‐ and group‐level decoding models. Importantly, group models outperform subject models on low‐accuracy subjects (although slightly impair high‐accuracy subjects) and can be helpful for initialising subject models. While we have not generally found group‐level models to perform better than subject‐level models, the performance of group modelling is expected to be even higher with bigger datasets. In order to provide physiological interpretation at the group level, we make use of permutation feature importance. This provides insights into the spatiotemporal and spectral information encoded in the models. All code is available on GitHub (https://github.com/ricsinaruto/MEG-group-decode).

Decoding individual words from non-invasive brain recordings across 723 participants

Decoding speech from non-invasive brain recordings

Decoding speech perception from non-invasive brain recordings

Brain2Word: Decoding Brain Activity for Language Generation

Deep Recurrent Encoder: A scalable end-to-end network to model brain signals

Speech decoding from stereo-electroencephalography (sEEG) signals using advanced deep learning methods

Decoding Continuous Character-based Language from Non-invasive Brain Recordings

Deep learning approaches for neural decoding across architectures and recording modalities

Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding

NeuSpeech: Decode Neural signal as Speech

Brain decoding: toward real-time reconstruction of visual perception

Semantic reconstruction of continuous language from non-invasive brain recordings

Group‐level brain decoding with deep learning

Group-level Brain Decoding with Deep Learning

The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning

Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

Semantic reconstruction of continuous language from MEG signals

Resolving Domain Shift For Representations Of Speech In Non-Invasive Brain Recordings

Challenges for cognitive decoding using deep learning methods

Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI