Abstract:Decoding linguistic information from non-invasive brain signals using EEG has gained increasing research attention due to its vast applicational potential. Recently, a number of works have adopted a generative-based framework to decode electroencephalogram (EEG) signals into sentences by utilizing the power generative capacity of pretrained large language models (LLMs). However, this approach has several drawbacks that hinder the further development of linguistic applications for brain-computer interfaces (BCIs). Specifically, the ability of the EEG encoder to learn semantic information from EEG data remains questionable, and the LLM decoder's tendency to generate sentences based on its training memory can be hard to avoid. These issues necessitate a novel approach for converting EEG signals into sentences. In this paper, we propose a novel two-step pipeline that addresses these limitations and enhances the validity of linguistic EEG decoding research. We first confirm that word-level semantic information can be learned from EEG data recorded during natural reading by training a Conformer encoder via a masked contrastive objective for word-level classification. To achieve sentence decoding results, we employ a training-free retrieval method to retrieve sentences based on the predictions from the EEG encoder. Extensive experiments and ablation studies were conducted in this paper for a comprehensive evaluation of the proposed approach. Visualization of the top prediction candidates reveals that our model effectively groups EEG segments into semantic categories with similar meanings, thereby validating its ability to learn patterns from unspoken EEG recordings. Despite the exploratory nature of this work, these results suggest that our method holds promise for providing more reliable solutions for converting EEG signals into text.

What problem does this paper attempt to address?

This paper attempts to solve several key problems encountered when decoding linguistic information from electroencephalogram (EEG) signals, especially during the process of converting EEG signals into sentences during silent reading tasks. Specifically: 1. **Insufficient Semantic Information Learning**: The existing EEG encoders are questionable in their ability to learn semantic information from EEG data. Traditional generation - based methods may not enable EEG encoders to truly learn to capture semantic patterns. 2. **The Influence of Pretrained Large Language Models (LLMs)**: Powerful pretrained language models (such as LLMs), when used as decoders, tend to generate sentences based on their training memory rather than relying on the input EEG signals. This leads to the neglect of EEG signals, causing EEG encoders to fail to effectively learn meaningful features. 3. **Data Sparsity and Noise Problems**: EEG signals in silent reading tasks have high data sparsity and noise, which increases the decoding difficulty. For example, participants may have inconsistent attention to different words when reading, thus affecting the decoding effect. To solve these problems, the authors propose a new two - step pipeline method, called EEG - to - Text Retrieval (ETER), aiming at: - **Verifying the Learning Ability of EEG Encoders**: Train a Conformer encoder to learn word - level semantic representations and optimize it using the masked contrastive loss function. This can ensure that EEG encoders can effectively extract semantic information from EEG data. - **Eliminating the Bias of Pretrained Language Models**: Adopt an untrained retrieval method to avoid the bias brought by relying on powerful pretrained language models. Specifically, find the most relevant sentences through the set of predicted keywords (SK) combined with the beam - search retrieval (BSR) method. - **Improving Decoding Accuracy**: Through a large number of experiments and ablation studies, the effectiveness of the proposed ETER method in word - level classification and sentence retrieval has been verified, achieving high precision and recall rates. In conclusion, the main contribution of this paper lies in providing a new and more reliable method for converting EEG signals into text, which solves several limitations in existing methods.

Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings

Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding

BELT: Bootstrapped EEG-to-language Training by Natural Language Supervision

Decoding Bilingual EEG Signals With Complex Semantics Using Adaptive Graph Attention Convolutional Network

Characterizing Neural Entrainment to Hierarchical Linguistic Units using Electroencephalography (EEG).

Neural Encoding and Decoding With Distributed Sentence Representations

Speech decoding from stereo-electroencephalography (sEEG) signals using advanced deep learning methods

Decoding Linguistic Representations of Human Brain

Language Generation from Brain Recordings

NeuSpeech: Decode Neural signal as Speech

Decoding speech from non-invasive brain recordings

A neural decoding algorithm that generates language from visual activity evoked by natural images

Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder

Identification of perceived sentences using deep neural networks in EEG

SEE: Semantically Aligned EEG-to-Text Translation

Understanding language-elicited EEG data by predicting it from a fine-tuned language model

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

EEG2TEXT: Open Vocabulary EEG-to-Text Decoding with EEG Pre-Training and Multi-View Transformer

BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation

Decoding human brain activity with deep learning

Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models