Abstract:Recent studies demonstrate the use of a two-stage supervised framework to generate images that depict human perception to visual stimuli from EEG, referring to EEG-visual reconstruction. They are, however, unable to reproduce the exact visual stimulus, since it is the human-specified annotation of images, not their data, that determines what the synthesized images are. Moreover, synthesized images often suffer from noisy EEG encodings and unstable training of generative models, making them hard to recognize. Instead, we present a single-stage EEG-visual retrieval paradigm where data of two modalities are correlated, as opposed to their annotations, allowing us to recover the exact visual stimulus for an EEG clip. We maximize the mutual information between the EEG encoding and associated visual stimulus through optimization of a contrastive self-supervised objective, leading to two additional benefits. One, it enables EEG encodings to handle visual classes beyond seen ones during training, since learning is not directed at class annotations. In addition, the model is no longer required to generate every detail of the visual stimulus, but rather focuses on cross-modal alignment and retrieves images at the instance level, ensuring distinguishable model output. Empirical studies are conducted on the largest single-subject EEG dataset that measures brain activities evoked by image stimuli. We demonstrate the proposed approach completes an instance-level EEG-visual retrieval task which existing methods cannot. We also examine the implications of a range of EEG and visual encoder structures. Furthermore, for a mostly studied semantic-level EEG-visual classification task, despite not using class annotations, the proposed method outperforms state-of-the-art supervised EEG-visual reconstruction approaches, particularly on the capability of open class recognition.

Visual Images Memory Recall Based on ClassRBM and Free Energy Minimization

A Biologically Inspired Visual Working Memory for Deep Networks

Classification and Generation of real-world data with an Associative Memory Model

A Visual Memory Information Interaction Mechanism Inspired Graph Neural Network for Highly Imbalanced Image Classification

Memory Encoding Model

Introducing Memory and Association Mechanism into a Biologically Inspired Visual Model

Learning Computational Models of Video Memorability from Fmri Brain Imaging

See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity

Brain Inspired Cognitive System for Learning and Memory

Neural Encoding for Image Recall: Human-Like Memory

Semantic memory-based dynamic neural network using memristive ternary CIM and CAM for 2D and 3D vision

Towards flexible perception with visual memory

A dynamic neural resource model bridges sensory and working memory

An Attention Selection Model With Visual Memory And Online Learning

A Memorization Learning Model For Image Retrieval

Learning Low-Rank Sparse Representations With Robust Relationship Inference for Image Memorability Prediction

Associative Memories via Predictive Coding

Visual Memory Neural Network for Artistic Graphic Design

Brain State Decoding for Rapid Image Retrieval

Defining Image Memorability using the Visual Memory Schema