Abstract:As it is true for human perception that we gather information from different sources in natural and multi-modality forms, learning from multi-modalities has become an effective scheme for various information retrieval problems. In this paper, we propose a novel multi-modality fusion approach for video search, where the search modalities are derived from a diverse set of knowledge sources, such as text transcript from speech recognition, low-level visual features from video frames, and high-level semantic visual concepts from supervised learning. Since the effectiveness of each search modality greatly depends on specific user queries, prompt determination of the importance of a modality to a user query is a critical issue in multi-modality search. Our proposed approach, named concept-driven multi-modality fusion (CDMF), explores a large set of predefined semantic concepts for computing multi-modality fusion weights in a novel way. Specifically, in CDMF, we decompose the query-modality relationship into two components that are much easier to compute: query-concept relatedness and concept-modality relevancy. The former can be efficiently estimated online using semantic and visual mapping techniques, while the latter can be computed offline based on concept detection accuracy of each modality. Such a decomposition facilitates the need of adaptive learning of fusion weights for each user query on-the-fly, in contrast to the existing approaches which mostly adopted predefined query classes and/or modality weights. Experimental results on TREC video-retrieval evaluation 2005-2008 dataset validate the effectiveness of our approach, which outperforms the existing multi-modality fusion methods and achieves near-optimal performance (from oracle fusion) for many test queries.

An Effective Video Retrieval Approach Based on Multi-modality Concept Correlation Graph

Graph-based Multi-Space Semantic Correlation Propagation for Video Retrieval

Explicit and implicit concept-based video retrieval with bipartite graph propagation model.

Video search re-ranking via multi-graph propagation.

Multi-Modal Relational Graph for Cross-Modal Video Moment Retrieval

Video retrieval with multi-modal features.

Concept-Driven Multi-Modality Fusion for Video Search

Learning Structured Concept-Segments for Interactive Video Retrieval

Learning a Multi-Concept Video Retrieval Model with Multiple Latent Variables

Text-Video Retrieval via Variational Multi-Modal Hypergraph Networks

The importance of query-concept-mapping for automatic video retrieval.

Query Representation by Structured Concept Threads with Application to Interactive Video Retrieval.

Spatial-temporal Graphs for Cross-modal Text2Video Retrieval

Multiple Hypergraph Ranking for Video Concept Detection

Multi-Modality Transfer Based on Multi-Graph Optimization for Domain Adaptive Video Concept Annotation

View-based 3D Object Retrieval Via Multi-Modal Graph Learning

Correlative Multi-Label Video Annotation.

Distribution-based Concept Selection for Concept-Based Video Retrieval.

Research on Video Retrieval Technology based on Multimodal Fusion and Attention Mechanism

Cross-Modal Learning Based on Semantic Correlation and Multi-Task Learning for Text-Video Retrieval

An integrated semantic-based approach in concept based video retrieval