Abstract:The explosive growth of digital video data renders a profound challenge to succinct, informative, and human-centric representations of video contents. This quickly-evolving research topic is typically called 'video abstraction'. We are motivated by the facts that the human brain is the end-evaluator of multimedia content and that the brain's responses can quantitatively reveal its attentional engagement in the comprehension of video. We propose a novel video abstraction paradigm which leverages functional magnetic resonance imaging (fMRI) to monitor and quantify the brain's responses to video stimuli. These responses are used to guide the extraction of visually informative segments from videos. Specifically, most relevant brain regions involved in video perception and cognition are identified to form brain networks. Then, the propensity for synchronization (PFS) derived from spectral graph theory is utilized over the brain networks to yield the benchmark attention curves based on the fMRI-measured brain responses to a number of training video streams. These benchmark attention curves are applied to guide and optimize the combinations of a variety of low-level visual features created by the Bayesian surprise model. In particular, in the training stage, the optimization objective is to ensure that the learned attentional model correlates well with the brain's responses and reflects the attention that viewers pay to video contents. In the application stage, the attention curves predicted by the learned and optimized attentional model serve as an effective benchmark to abstract testing videos. Evaluations on a set of video sequences from the TRECVID database demonstrate the effectiveness of the proposed framework.

Refinement of Extracted Visual Attention Areas in Video Sequences

Key frame extraction based on the visual attention model for lane surveillance video

Visual Attention Based Video Object Segmentation in MPEG Compressed Domain

Video Abstraction via Attention Model and On-Line Clustering

Video abstraction based on the visual attention model and online clustering

Local Attention Sequence Model for Video Object Detection

Robust video fingerprinting based on visual attention regions

A Video Image Compression Method Based on Visually Salient Features.

An Approach to Video Object Extraction in the Compressed Domain

A novel attention model and its application in video analysis

Video Abstraction Based on Fmri-Driven Visual Attention Model

Spatial and Temporal Visual Attention Prediction in Videos Using Eye Movement Data

Video Question Answering Via Gradually Refined Attention over Appearance and Motion

A Distortion-Weighing Spatiotemporal Visual Attention Model for Video Analysis

Using Spatial‐Temporal Attention for Video Quality Evaluation

Multichannel Attention Refinement for Video Question Answering.

Video Saliency Detection via Dynamic Consistent Spatio-Temporal Attention Modelling.

Region-based visual attention analysis with its application in image browsing on small displays.

Video image assessment with a distortion-weighing spatiotemporal visual attention model

A VISUAL ATTENTION MODEL FOR NATURAL SCENES BASED ON DYNAMIC FEATURE COMBINATION

Compressive video sensing based on user attention model