Abstract:As it is true for human perception that we gather information from different sources in natural and multi-modality forms, learning from multi-modalities has become an effective scheme for various information retrieval problems. In this paper, we propose a novel multi-modality fusion approach for video search, where the search modalities are derived from a diverse set of knowledge sources, such as text transcript from speech recognition, low-level visual features from video frames, and high-level semantic visual concepts from supervised learning. Since the effectiveness of each search modality greatly depends on specific user queries, prompt determination of the importance of a modality to a user query is a critical issue in multi-modality search. Our proposed approach, named concept-driven multi-modality fusion (CDMF), explores a large set of predefined semantic concepts for computing multi-modality fusion weights in a novel way. Specifically, in CDMF, we decompose the query-modality relationship into two components that are much easier to compute: query-concept relatedness and concept-modality relevancy. The former can be efficiently estimated online using semantic and visual mapping techniques, while the latter can be computed offline based on concept detection accuracy of each modality. Such a decomposition facilitates the need of adaptive learning of fusion weights for each user query on-the-fly, in contrast to the existing approaches which mostly adopted predefined query classes and/or modality weights. Experimental results on TREC video-retrieval evaluation 2005-2008 dataset validate the effectiveness of our approach, which outperforms the existing multi-modality fusion methods and achieves near-optimal performance (from oracle fusion) for many test queries.

A Hybrid Approach to Video Retrieval in a Generic Video Management and Application Processing Framework

Accommodating hybrid retrieval in a comprehensive video database management system

Web-based Video Database Management: Issues, Mechanisms, and Experimental Development

Development of A Web-Based Video Management and Application Processing System

Hierarchical Video Data Modeling and Indexing for Virtual Scene Construction

Video retrieval with multi-modal features.

Content-based Video Retrieval Integrating Human Perception.

A Tree-Based Paradigm for Content-Based Video Retrieval and Management

Strategies for Searching Video Content with Text Queries or Video Examples.

Videobase - A Prototype Of Video Database Managing System

VideoMap: an interactive video retrieval system of MCG-ICT-CAS

Concept-Driven Multi-Modality Fusion for Video Search

A MULTI-CHANNEL RETRIEVAL SYSTEM FOR MULTIMEDIA DOCUMENTS

Semantic-based surveillance video retrieval

A Unified Framework for Video Summarization, Browsing, and Retrieval

ClassView: Hierarchical Video Shot Classification, Indexing, and Accessing

[Invited Paper] Strategies for Searching Video Content with Text Queries or Video Examples: Features, Semantic Detectors, Fusion, Efficient Search and Reranking

Multimedia Analysis and Retrieval System

Video diver: generic video indexing with diverse features.

InsightVideo: toward hierarchical video content organization for efficient browsing, summarization and retrieval

An Efficient Approach to Content-Based Object Retrieval in Videos.