Abstract:Recently, electroencephalography (EEG) signals have been actively incorporated to decode brain activity to visual or textual stimuli and achieve object recognition in multi-modal AI. Accordingly, endeavors have been focused on building EEG-based datasets from visual or textual single-modal stimuli. However, these datasets offer limited EEG epochs per category, and the complex semantics of stimuli presented to participants compromise their quality and fidelity in capturing precise brain activity. The study in neuroscience unveils that the relationship between visual and textual stimulus in EEG recordings provides valuable insights into the brain's ability to process and integrate multi-modal information simultaneously. Inspired by this, we propose a novel large-scale multi-modal dataset, named EIT-1M, with over 1 million EEG-image-text pairs. Our dataset is superior in its capacity of reflecting brain activities in simultaneously processing multi-modal information. To achieve this, we collected data pairs while participants viewed alternating sequences of visual-textual stimuli from 60K natural images and category-specific texts. Common semantic categories are also included to elicit better reactions from participants' brains. Meanwhile, response-based stimulus timing and repetition across blocks and sessions are included to ensure data diversity. To verify the effectiveness of EIT-1M, we provide an in-depth analysis of EEG data captured from multi-modal stimuli across different categories and participants, along with data quality scores for transparency. We demonstrate its validity on two tasks: 1) EEG recognition from visual or textual stimuli or both and 2) EEG-to-visual generation.

A large-scale fMRI dataset for the visual processing of naturalistic scenes

A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence

An open-access dataset of naturalistic viewing using simultaneous EEG-fMRI

BOLD5000: A public fMRI dataset of 5000 images

A large and rich EEG dataset for modeling human visual object recognition

An fMRI dataset in response to large-scale short natural dynamic facial expression videos

A multimodal fMRI dataset unifying naturalistic processes with a rich array of experimental tasks

A naturalistic neuroimaging database for understanding the brain using ecological stimuli

A large calcium-imaging dataset reveals a systematic V4 organization for natural scenes

Visualizing fMRI BOLD responses to diverse naturalistic scenes using retinotopic projection

fNIRS Dataset During Complex Scene Analysis

DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision

A studyforrest extension, retinotopic mapping and localization of higher visual areas

A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction

The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes

A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex

Source imaging of high-density visual evoked potentials with multi-scale brain parcellations and connectomes

Modeling short visual events through the BOLD moments video fMRI dataset and metadata

A Studyforrest Extension, MEG Recordings While Watching the Audio-Visual Movie “forrest Gump”

EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More

BOLD Moments: modeling short visual events through a video fMRI dataset and metadata