Abstract:Objective. Electroencephalography (EEG) is a widely used technology for recording brain activity in brain-computer interface (BCI) research, where understanding the encoding-decoding relationship between stimuli and neural responses is a fundamental challenge. Recently, there is a growing interest in encoding-decoding natural stimuli in a single-trial setting, as opposed to traditional BCI literature where multi-trial presentations of synthetic stimuli are commonplace. While EEG responses to natural speech have been extensively studied, such stimulus-following EEG responses to natural video footage remain underexplored. Approach. We collect a new EEG dataset with subjects passively viewing a film clip and extract a few video features that have been found to be temporally correlated with EEG signals. However, our analysis reveals that these correlations are mainly driven by shot cuts in the video. To avoid the confounds related to shot cuts, we construct another EEG dataset with natural single-shot videos as stimuli and propose a new set of object-based features. Main Results. We demonstrate that previous video features lack robustness in capturing the coupling with EEG signals in the absence of shot cuts, and that the proposed object-based features exhibit significantly higher correlations. Furthermore, we show that the correlations obtained with these proposed features are not dominantly driven by eye movements. Additionally, we quantitatively verify the superiority of the proposed features in a match-mismatch (MM) task. Finally, we evaluate to what extent these proposed features explain the variance in coherent stimulus responses across subjects. Significance. This work provides valuable insights into feature design for video-EEG analysis and paves the way for applications such as visual attention decoding.

Modeling short visual events through the BOLD moments video fMRI dataset and metadata

BOLD Moments: modeling short visual events through a video fMRI dataset and metadata

An open-access dataset of naturalistic viewing using simultaneous EEG-fMRI

Multimodal single-neuron, intracranial EEG, and fMRI brain responses during movie watching in human patients

BOLD5000: A public fMRI dataset of 5000 images

NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

Learning Computational Models of Video Memorability from Fmri Brain Imaging

Visualizing fMRI BOLD responses to diverse naturalistic scenes using retinotopic projection

A large and rich EEG dataset for modeling human visual object recognition

Video Abstraction Based on Fmri-Driven Visual Attention Model

Identifying Naturalistic Movies from Human Brain Activity with High-Density Diffuse Optical Tomography

A multimodal fMRI dataset unifying naturalistic processes with a rich array of experimental tasks

A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence

Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity

MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

An fMRI dataset in response to large-scale short natural dynamic facial expression videos

The Visual Experience Dataset: Over 200 Recorded Hours of Integrated Eye Movement, Odometry, and Egocentric Video

Identifying temporal correlations between natural single-shot videos and EEG signals

Encoding functional brain interactions from computational visual features

Neural Representations of Dynamic Visual Stimuli

Neural Processing of Naturalistic Audiovisual Events in Space and Time