Abstract:As it is true for human perception that we gather information from different sources in natural and multi-modality forms, learning from multi-modalities has become an effective scheme for various information retrieval problems. In this paper, we propose a novel multi-modality fusion approach for video search, where the search modalities are derived from a diverse set of knowledge sources, such as text transcript from speech recognition, low-level visual features from video frames, and high-level semantic visual concepts from supervised learning. Since the effectiveness of each search modality greatly depends on specific user queries, prompt determination of the importance of a modality to a user query is a critical issue in multi-modality search. Our proposed approach, named concept-driven multi-modality fusion (CDMF), explores a large set of predefined semantic concepts for computing multi-modality fusion weights in a novel way. Specifically, in CDMF, we decompose the query-modality relationship into two components that are much easier to compute: query-concept relatedness and concept-modality relevancy. The former can be efficiently estimated online using semantic and visual mapping techniques, while the latter can be computed offline based on concept detection accuracy of each modality. Such a decomposition facilitates the need of adaptive learning of fusion weights for each user query on-the-fly, in contrast to the existing approaches which mostly adopted predefined query classes and/or modality weights. Experimental results on TREC video-retrieval evaluation 2005-2008 dataset validate the effectiveness of our approach, which outperforms the existing multi-modality fusion methods and achieves near-optimal performance (from oracle fusion) for many test queries.

OCTOPUS: aggressive search of multi-modality data using multifaceted knowledge base.

Search for Multi-modality Data in Digital Libraries

Web-Based Multimedia Retrieval: Balancing out Between Common Knowledge and Personalized Views

Multi-Modal Retrieval for Multimedia Digital Libraries: Issues, Architecture, and Mechanisms

MPVR: a multi-perspective visual retrieval toolkit for multi-dimensional data.

Intelligent Search on Integrated Knowledge Base of Traditional Chinese Medicine

Consensus Knowledge Exploitation for Partial Query Based Image Retrieval

Multimedia Analysis and Retrieval System

Understanding Multimedia Document Semantics for Cross-Media Retrieval

Knowledge Ontology Establishment of Collaborative and Application

A MULTI-CHANNEL RETRIEVAL SYSTEM FOR MULTIMEDIA DOCUMENTS

An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

Effective Deep Learning-Based Multi-Modal Retrieval

Cross-Media Retrieval: Concepts, Advances And Challenges

Effective Multi-Modal Retrieval Based on Stacked Auto-Encoders

MUST: an Effective and Scalable Framework for Multimodal Search of Target Modality

News Video Retrieval By Learning Multimodal Semantic Information

The Research of Multimedia Cross Reference Retrieval System

Know2Look: Commonsense Knowledge for Visual Search

Concept-Driven Multi-Modality Fusion for Video Search

A Survey of Multimodal Composite Editing and Retrieval