Abstract:As it is true for human perception that we gather information from different sources in natural and multi-modality forms, learning from multi-modalities has become an effective scheme for various information retrieval problems. In this paper, we propose a novel multi-modality fusion approach for video search, where the search modalities are derived from a diverse set of knowledge sources, such as text transcript from speech recognition, low-level visual features from video frames, and high-level semantic visual concepts from supervised learning. Since the effectiveness of each search modality greatly depends on specific user queries, prompt determination of the importance of a modality to a user query is a critical issue in multi-modality search. Our proposed approach, named concept-driven multi-modality fusion (CDMF), explores a large set of predefined semantic concepts for computing multi-modality fusion weights in a novel way. Specifically, in CDMF, we decompose the query-modality relationship into two components that are much easier to compute: query-concept relatedness and concept-modality relevancy. The former can be efficiently estimated online using semantic and visual mapping techniques, while the latter can be computed offline based on concept detection accuracy of each modality. Such a decomposition facilitates the need of adaptive learning of fusion weights for each user query on-the-fly, in contrast to the existing approaches which mostly adopted predefined query classes and/or modality weights. Experimental results on TREC video-retrieval evaluation 2005-2008 dataset validate the effectiveness of our approach, which outperforms the existing multi-modality fusion methods and achieves near-optimal performance (from oracle fusion) for many test queries.

Research on Video Retrieval Technology based on Multimodal Fusion and Attention Mechanism

Fine-Grained Cross-Modal Retrieval with Triple-Streamed Memory Fusion Transformer Encoder

Coarse-to-fine dual-level attention for video-text cross modal retrieval

Video retrieval with multi-modal features.

A Multi-interaction Model with Cross-Branch Feature Fusion for Video-Text Retrieval.

Dynamic Multimodal Fusion in Video Search

Feature Fusion Based on Transformer for Cross-modal Retrieval

Joint Feature Optimization and Fusion for Compressed Action Recognition

Concept-Driven Multi-Modality Fusion for Video Search

Multimodal Fusion Method Based on Self-Attention Mechanism

Attention-Based Multimodal Fusion for Video Description

Cross-modal Search Method of Technology Video based on Adversarial Learning and Feature Fusion

News Video Retrieval By Learning Multimodal Semantic Information

A multi-modal fusion approach for measuring web video relatedness

Multi-modal recommendation algorithm fusing visual and textual features

Audio-Visual Fusion Based on Interactive Attention for Person Verification

Deep Relationship Analysis in Video with Multimodal Feature Fusion

Multimodal Fusion for Video Search Reranking

Efficient Heuristic Methods for Multimodal Fusion and Concept Fusion in Video Concept Detection

A Video Target Re-Recognition Method Based on Adaptive Attention Enhancement and Multi-Scale Feature Fusion

SAM: Modeling Scene, Object and Action with Semantics Attention Modules for Video Recognition