Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding

Hamza Amrani,Daniela Micucci,Paolo Napoletano

2023-11-15

Abstract:Previous research has demonstrated the potential of using pre-trained language models for decoding open vocabulary Electroencephalography (EEG) signals captured through a non-invasive Brain-Computer Interface (BCI). However, the impact of embedding EEG signals in the context of language models and the effect of subjectivity, remain unexplored, leading to uncertainty about the best approach to enhance decoding performance. Additionally, current evaluation metrics used to assess decoding effectiveness are predominantly syntactic and do not provide insights into the comprehensibility of the decoded output for human understanding. We present an end-to-end deep learning framework for non-invasive brain recordings that brings modern representational learning approaches to neuroscience. Our proposal introduces the following innovations: 1) an end-to-end deep learning architecture for open vocabulary EEG decoding, incorporating a subject-dependent representation learning module for raw EEG encoding, a BART language model, and a GPT-4 sentence refinement module; 2) a more comprehensive sentence-level evaluation metric based on the BERTScore; 3) an ablation study that analyses the contributions of each module within our proposal, providing valuable insights for future research. We evaluate our approach on two publicly available datasets, ZuCo v1.0 and v2.0, comprising EEG recordings of 30 subjects engaged in natural reading tasks. Our model achieves a BLEU-1 score of 42.75%, a ROUGE-1-F of 33.28%, and a BERTScore-F of 53.86%, outperforming the previous state-of-the-art methods by 3.38%, 8.43%, and 6.31%, respectively.

Signal Processing,Computation and Language,Human-Computer Interaction,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address several key issues in decoding non-invasive electroencephalogram (EEG) signals into open vocabulary text:

1. **Improving Decoding Performance**:
- Current research has demonstrated the potential of using pre-trained language models to decode open vocabulary EEG signals, but the impact of embedding EEG signals in language models and the influence of individual differences have not been fully explored.
- Existing evaluation metrics mainly focus on grammatical structure and fail to adequately reflect the human comprehensibility of the decoded output.

2. **Introducing Innovative Methods**:
- Proposes an end-to-end deep learning framework for open vocabulary decoding of non-invasive EEG signals.
- Introduces a subject-dependent representation learning module based on the BART language model and a GPT-4 sentence optimization module.
- Uses BERTScore as a more comprehensive sentence-level evaluation metric to better reflect semantic similarity.

3. **Analyzing Module Contributions**:
- Conducts ablation studies to analyze the contribution of each module to overall performance, providing valuable insights for future research.

4. **Validating Method Effectiveness**:
- Conducts extensive evaluations on two public datasets, ZuCo v1.0 and v2.0, which contain EEG recordings of 30 subjects during natural reading tasks.
- Experimental results show that the model outperforms existing state-of-the-art methods by 3.38%, 8.43%, and 6.31% on BLEU-1, ROUGE-1-F, and BERTScore-F metrics, respectively.

### Summary

The main goal of the paper is to improve the performance of decoding non-invasive EEG signals into open vocabulary text by introducing a new end-to-end deep learning framework that combines pre-trained language models and subject-dependent representation learning modules. The paper also validates the effectiveness of the approach through more comprehensive evaluation metrics. Through ablation studies, the paper explores the contribution of each module to overall performance, providing important references for future research.