Abstract:Decoding linguistic information from non-invasive brain signals using EEG has gained increasing research attention due to its vast applicational potential. Recently, a number of works have adopted a generative-based framework to decode electroencephalogram (EEG) signals into sentences by utilizing the power generative capacity of pretrained large language models (LLMs). However, this approach has several drawbacks that hinder the further development of linguistic applications for brain-computer interfaces (BCIs). Specifically, the ability of the EEG encoder to learn semantic information from EEG data remains questionable, and the LLM decoder's tendency to generate sentences based on its training memory can be hard to avoid. These issues necessitate a novel approach for converting EEG signals into sentences. In this paper, we propose a novel two-step pipeline that addresses these limitations and enhances the validity of linguistic EEG decoding research. We first confirm that word-level semantic information can be learned from EEG data recorded during natural reading by training a Conformer encoder via a masked contrastive objective for word-level classification. To achieve sentence decoding results, we employ a training-free retrieval method to retrieve sentences based on the predictions from the EEG encoder. Extensive experiments and ablation studies were conducted in this paper for a comprehensive evaluation of the proposed approach. Visualization of the top prediction candidates reveals that our model effectively groups EEG segments into semantic categories with similar meanings, thereby validating its ability to learn patterns from unspoken EEG recordings. Despite the exploratory nature of this work, these results suggest that our method holds promise for providing more reliable solutions for converting EEG signals into text.

Neural Memory Decoding with EEG Data and Representation Learning

EEG decoders track memory dynamics

Decoding Working Memory Representation Based on Single-Trial EEG by a Deep Learning Framework

Decoding EEG for optimizing naturalistic memory

Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding

Decoding working memory-related information from repeated psychophysiological EEG experiments using convolutional and contrastive neural networks

Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks

Learning Exemplar Representations in Single-Trial EEG Category Decoding

EEG decoding with conditional identification information

Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models

Decoding Natural Images from EEG for Object Recognition

Neural representations of predicted events: Evidence from time-resolved EEG decoding

Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings

Neural decoding of music from the EEG

Decoding Imagined Speech from EEG Data: A Hybrid Deep Learning Approach to Capturing Spatial and Temporal Features

Bayesian Neural Decoding Using A Diversity-Encouraging Latent Representation Learning Method

Deep learning approaches for neural decoding across architectures and recording modalities

Image classification and reconstruction from low-density EEG

Evaluating the structure of cognitive tasks with transfer learning

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding