Abstract:Recent studies demonstrate the use of a two-stage supervised framework to generate images that depict human perception to visual stimuli from EEG, referring to EEG-visual reconstruction. They are, however, unable to reproduce the exact visual stimulus, since it is the human-specified annotation of images, not their data, that determines what the synthesized images are. Moreover, synthesized images often suffer from noisy EEG encodings and unstable training of generative models, making them hard to recognize. Instead, we present a single-stage EEG-visual retrieval paradigm where data of two modalities are correlated, as opposed to their annotations, allowing us to recover the exact visual stimulus for an EEG clip. We maximize the mutual information between the EEG encoding and associated visual stimulus through optimization of a contrastive self-supervised objective, leading to two additional benefits. One, it enables EEG encodings to handle visual classes beyond seen ones during training, since learning is not directed at class annotations. In addition, the model is no longer required to generate every detail of the visual stimulus, but rather focuses on cross-modal alignment and retrieves images at the instance level, ensuring distinguishable model output. Empirical studies are conducted on the largest single-subject EEG dataset that measures brain activities evoked by image stimuli. We demonstrate the proposed approach completes an instance-level EEG-visual retrieval task which existing methods cannot. We also examine the implications of a range of EEG and visual encoder structures. Furthermore, for a mostly studied semantic-level EEG-visual classification task, despite not using class annotations, the proposed method outperforms state-of-the-art supervised EEG-visual reconstruction approaches, particularly on the capability of open class recognition.

Capturing Pictures from Human Vision Using SSVEP and Lock-in Amplifier

Painting with the Eye: Understanding the Visual Field of the Human Eye with SSVEP.

Eye Blink Artifact Detection Based on Multi-Dimensional EEG Feature Fusion and Optimization.

Interpreting the Visual Acuity of the Human Eye with Wearable EEG Device and SSVEP

Eye-Moving Assisted Eeg Acquisition and Eeg Feature Detection

Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals

Unveiling Visual Physiology and Steady-State Evoked Potentials using Low-Cost and Transferable Electroencephalography for Evaluating Neuronal Activation

Steady State Visually Evoked Potentials detection using a single electrode consumer-grade EEG device for BCI applications

A Single-Channel Consumer-Grade EEG Device for Brain-Computer Interface: Enhancing Detection of SSVEP and Its Amplitude Modulation

SSVEP Harmonic Fusion for Improved Visual Field Reconstruction with CNN

eyeSay: Brain Visual Dynamics Decoding With Deep Learning & Edge Computing

Human Recognition Using Deep Neural Networks and Spatial Patterns of SSVEP Signals

A Spiking Artificial Vision Architecture Based on Fully Emulating the Human Vision

The VEP Booster: A Closed-Loop AI System for Visual EEG Biomarker Auto-generation

Processing and Analysis of EEG Signal for SSVEP Detection

Reconstructing Visual Stimulus Images from EEG Signals Based on Deep Visual Representation Model

SSVEP-EEG Feature Enhancement Method Using an Image Sharpening Filter

See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity

Research on EEG Feature Decoding Based on Stimulus Image

Eye gaze tracking based on analysis of electrooculography and steady-state visual evoked potentials

Applying advanced machine learning models to classify electro-physiological activity of human brain for use in biometric identification