Abstract:Recent studies demonstrate the use of a two-stage supervised framework to generate images that depict human perception to visual stimuli from EEG, referring to EEG-visual reconstruction. They are, however, unable to reproduce the exact visual stimulus, since it is the human-specified annotation of images, not their data, that determines what the synthesized images are. Moreover, synthesized images often suffer from noisy EEG encodings and unstable training of generative models, making them hard to recognize. Instead, we present a single-stage EEG-visual retrieval paradigm where data of two modalities are correlated, as opposed to their annotations, allowing us to recover the exact visual stimulus for an EEG clip. We maximize the mutual information between the EEG encoding and associated visual stimulus through optimization of a contrastive self-supervised objective, leading to two additional benefits. One, it enables EEG encodings to handle visual classes beyond seen ones during training, since learning is not directed at class annotations. In addition, the model is no longer required to generate every detail of the visual stimulus, but rather focuses on cross-modal alignment and retrieves images at the instance level, ensuring distinguishable model output. Empirical studies are conducted on the largest single-subject EEG dataset that measures brain activities evoked by image stimuli. We demonstrate the proposed approach completes an instance-level EEG-visual retrieval task which existing methods cannot. We also examine the implications of a range of EEG and visual encoder structures. Furthermore, for a mostly studied semantic-level EEG-visual classification task, despite not using class annotations, the proposed method outperforms state-of-the-art supervised EEG-visual reconstruction approaches, particularly on the capability of open class recognition.

A Visual EEG Paradigm and Dataset for Recognizing the Size Transformation of Images

A large and rich EEG dataset for modeling human visual object recognition

An Open Dataset for Wearable SSVEP-Based Brain-Computer Interfaces

EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More

Decoding Human Visual Colour EEG Information Using Machine Learning and Visual Evoked Potentials

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

A Benchmark Dataset for SSVEP-Based Brain–Computer Interfaces

An EEG Dataset for Studying Asynchronous Steady-State Visual Evoked Potential (SSVEP) Based Brain Computer Interfaces

Dual-Alpha: a Large EEG Study for Dual-Frequency SSVEP Brain-Computer Interface.

Attention-Based Parallel Multiscale Convolutional Neural Network for Visual Evoked Potentials EEG Classification.

Decoding Visual Recognition of Objects from EEG Signals based on Attention-Driven Convolutional Neural Network

Decoding Brain Signals from Rapid-Event EEG for Visual Analysis Using Deep Learning

Alljoined1 -- A dataset for EEG-to-Image decoding

A Benchmark Dataset for RSVP-Based Brain–Computer Interfaces

A Large Dataset for VEP Based Brain-Computer Interfaces Employing Narrow-Band Code Modulation and Frequency-Phase Modulation

Exploring the Feasibility of Single-Frequency Multi-Target SSVEP-Based BCI for Online Control

See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity

Dataset Evaluation Method and Application for Performance Testing of SSVEP-BCI Decoding Algorithm

Research on EEG Feature Decoding Based on Stimulus Image

EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy

A Linear-Attention-combined Convolutional Neural Network for EEG-based Visual Stimulus Recognition