Abstract:Functional magnetic resonance imaging (fMRI) is a powerful tool to probe the human [email protected]?s perception and cognition. Besides being extensively exploited in the clinical applications, fMRI technique is also useful to [email protected]?s ordinary life. In this paper, we investigate a novel application of leveraging fMRI techniques to video clustering and retrieval. In the proposed work, we successfully integrate semantic human-centric features derived from natural stimulus fMRI data and low-level visual-audio features to facilitate video clustering and retrieval, which is a significant innovation compared to the previous works relying on either fMRI-derived features or low-level visual-audio features. Our system consists of several algorithmic modules. First, fMRI data when the subjects are watching video shot samples are acquired. Then a newly developed brain networks localization system is employed to locate the cortical regions of interests (ROIs) for each individual subject. The functional interactions computed by wavelet transform coherence are quantified, from which the human-centric features are derived. Afterwards, the Gaussian process regression model mapping visual-audio feature space to an fMRI-derived feature space is trained, given the training samples. The trained model is then adopted to predict fMRI-derived features for videos without the fMRI data. Finally, the multi-modal spectral clustering and multi-modal ranking algorithm are adopted and proposed to integrate these two heterogeneous features for video clustering and retrieval, respectively. Our experiment on TRECVID database has demonstrated the precision of video clustering and retrieval can be substantially improved by integration of visual-audio features and fMRI-derived features.

A Novel Video Retrieval Method Based on Web Community Extraction Using Features of Video Materials

A Novel Approach to Text Detection and Extraction from Videos by Discriminative Features and Density

Leveraging Collective Wisdom for Web Video Retrieval Through Heterogeneous Community Discovery

A multi-modal fusion approach for measuring web video relatedness

Content-Based Video Retrieval By Example Clip On Www

RELEVANCE FEEDBACK OF VIDEO RETRIEVAL

Hierarchical feedback algorithm based on visual community discovery for interactive video retrieval

A Novel System for Video Retrieval of Frontal-View Indoor Moving Pedestrians

Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA

A New Approach to Retrieve Video by Example Video Clip

Content-based Video Retrieval Integrating Human Perception.

Web Video Recommendation and Long Tail Discovering

Video retrieval with multi-modal features.

Rank-mediated collaborative tagging recommendation service using video-tag relationship prediction

Towards content-based relevance ranking for video search.

Towards Hierarchical Context: Unfolding Visual Community Potential for Interactive Video Retrieval

An Effective Video Retrieval Approach Based on Multi-modality Concept Correlation Graph

A Novel Approach to Video Indexing and Retrieval Using Motion Activity

Clustering and retrieval of video shots based on natural stimulus fMRI

Utilizing Related Samples to Enhance Interactive Concept-Based Video Search

AdVR: linking ad video with products or service