Abstract:The explosive growth of digital video data renders a profound challenge to succinct, informative, and human-centric representations of video contents. This quickly-evolving research topic is typically called 'video abstraction'. We are motivated by the facts that the human brain is the end-evaluator of multimedia content and that the brain's responses can quantitatively reveal its attentional engagement in the comprehension of video. We propose a novel video abstraction paradigm which leverages functional magnetic resonance imaging (fMRI) to monitor and quantify the brain's responses to video stimuli. These responses are used to guide the extraction of visually informative segments from videos. Specifically, most relevant brain regions involved in video perception and cognition are identified to form brain networks. Then, the propensity for synchronization (PFS) derived from spectral graph theory is utilized over the brain networks to yield the benchmark attention curves based on the fMRI-measured brain responses to a number of training video streams. These benchmark attention curves are applied to guide and optimize the combinations of a variety of low-level visual features created by the Bayesian surprise model. In particular, in the training stage, the optimization objective is to ensure that the learned attentional model correlates well with the brain's responses and reflects the attention that viewers pay to video contents. In the application stage, the attention curves predicted by the learned and optimized attentional model serve as an effective benchmark to abstract testing videos. Evaluations on a set of video sequences from the TRECVID database demonstrate the effectiveness of the proposed framework.

BOLD Moments: modeling short visual events through a video fMRI dataset and metadata

Modeling short visual events through the BOLD moments video fMRI dataset and metadata

BOLD5000: A public fMRI dataset of 5000 images

An open-access dataset of naturalistic viewing using simultaneous EEG-fMRI

Multimodal single-neuron, intracranial EEG, and fMRI brain responses during movie watching in human patients

NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

Video Abstraction Based on Fmri-Driven Visual Attention Model

Visualizing fMRI BOLD responses to diverse naturalistic scenes using retinotopic projection

A multimodal fMRI dataset unifying naturalistic processes with a rich array of experimental tasks

Learning Computational Models of Video Memorability from Fmri Brain Imaging

A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence

A large and rich EEG dataset for modeling human visual object recognition

Identifying Naturalistic Movies from Human Brain Activity with High-Density Diffuse Optical Tomography

An fMRI dataset in response to large-scale short natural dynamic facial expression videos

MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

The Visual Experience Dataset: Over 200 Recorded Hours of Integrated Eye Movement, Odometry, and Egocentric Video

The magic, memory, and curiosity fMRI dataset of people viewing magic tricks

Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity

Dynamic representation of multidimensional object properties in the human brain

The Surprising Effectiveness of Multimodal Large Language Models for Video Moment Retrieval

A high-density diffuse optical tomography dataset of naturalistic viewing