Abstract:Extracting semantically consistent representations from multi-modal data helps computers understand the human world more comprehensively. Visual-semantic matching, as one of the fundamental tasks for multi-modal learning, attracts continuous attention. Recent research makes unflagging endeavors to enhance the matching performance, but sometimes at the expense of overlooking the delicate balance between efficiency and effectiveness. In this paper, we aim to address this dilemma through a newly proposed attention-mechanism-based architecture. To ensure optimal effectiveness, we adopt a more advanced Transformer Encoder (TE) as our basic model and introduce two significant ameliorations to tailor it for the visual-semantic matching task. Initially, we incorporate fine-grained supervision into the classic TE, allowing our model to capture sophisticated correspondences between different modalities. Subsequently, we employ a dynamic attention-evolving strategy to selectively pass useful information and strengthen the attention pattern consistency between adjacent TE blocks. To maintain efficiency, we propose a novel Select & Re-rank strategy that enables the model to ignore redundant information. This approach significantly reduces the computational cost and increases the matching speed with relatively minimal performance degradation. The proposed architecture can gradually capture and reorganize useful information from inter-modality as well as intra-modality under the supervision of both fine-grained and global similarity, which leads to more comprehensive and discriminative embeddings. Experiments on two benchmark datasets show that the proposed method achieves competitive results in terms of both efficiency and effectiveness.

Optimizing Multimodal Reranking for Web Image Search

Multimodal graph-based reranking for web image search.

Contextual Modeling on Auxiliary Points for Robust Image Reranking

Select & Re-Rank: Effectively and Efficiently Matching Multimodal Data with Dynamically Evolving Attention

Image Search Reranking with Query-Dependent Click-Based Relevance Feedback.

Visual Reranking through Weakly Supervised Multi-graph Learning

An Attribute-Assisted Reranking Model for Web Image Search.

Multimodal Fusion for Video Search Reranking

Click-boosting multi-modality graph-based reranking for image search

A Novel and Adaptive Method for Image Search Reranking

Exploiting Click Constraints and Multi-view Features for Image Re-ranking

An Adaptive Query Prototype Modeling Method for Image Search Reranking

Attribute-assisted Reranking for Web Image Retrieval

Learning to rerank images with enhanced spatial verification

Multimedia search reranking

Semantic Pre-alignment and Ranking Learning with Unified Framework for Cross-modal Retrieval

Advancing Re-Ranking with Multimodal Fusion and Target-Oriented Auxiliary Tasks in E-Commerce Search

Effective Deep Learning-Based Multi-Modal Retrieval

Dynamic Visual Semantic Sub-Embeddings and Fast Re-Ranking

Leveraging Exemplar and Saliency Model for Image Search Reranking

Multimedia search reranking: A literature survey