Abstract:Extracting semantically consistent representations from multi-modal data helps computers understand the human world more comprehensively. Visual-semantic matching, as one of the fundamental tasks for multi-modal learning, attracts continuous attention. Recent research makes unflagging endeavors to enhance the matching performance, but sometimes at the expense of overlooking the delicate balance between efficiency and effectiveness. In this paper, we aim to address this dilemma through a newly proposed attention-mechanism-based architecture. To ensure optimal effectiveness, we adopt a more advanced Transformer Encoder (TE) as our basic model and introduce two significant ameliorations to tailor it for the visual-semantic matching task. Initially, we incorporate fine-grained supervision into the classic TE, allowing our model to capture sophisticated correspondences between different modalities. Subsequently, we employ a dynamic attention-evolving strategy to selectively pass useful information and strengthen the attention pattern consistency between adjacent TE blocks. To maintain efficiency, we propose a novel Select & Re-rank strategy that enables the model to ignore redundant information. This approach significantly reduces the computational cost and increases the matching speed with relatively minimal performance degradation. The proposed architecture can gradually capture and reorganize useful information from inter-modality as well as intra-modality under the supervision of both fine-grained and global similarity, which leads to more comprehensive and discriminative embeddings. Experiments on two benchmark datasets show that the proposed method achieves competitive results in terms of both efficiency and effectiveness.

Automatic Matching Tool Selection Using Relevance Feedback In Mars

Select & Re-Rank: Effectively and Efficiently Matching Multimodal Data with Dynamically Evolving Attention

Online Metric Learning for Relevance Feedback in E-Commerce Image Retrieval Applications

Relevance Feedback: a Power Tool for Interactive Content-Based Image Retrieval

Relevance Feedback For Keyword And Visual Feature-Based Image Retrieval

A Relevance Feedback Architecture for Content-Based Multimedia Information Retrieval Systems

Relevance Feedback for Visual Data Retrieval

Relevance Feedback Techniques In Interactive Content-Based Image Retrieval

A Survey on Relevance Feedback Techniques in Content-based Medical Image Retrieval

Asymmetric Bagging and Random Subspace for Support Vector Machines-Based Relevance Feedback in Image Retrieval

An Effective and Simple Relevance Feedback Algorithm for Image Retrieval

Similarity-based Online Feature Selection in Content-Based Image Retrieval

Improving feature matching strategies for efficient image retrieval

Robust feature selection with self-matching score

Supporting Similarity Queries in MARS

A Powerful Relevance Feedback Mechanism for Content-Based 3D Model Retrieval

Relevance Feedback in Content-Based Image Retrieval: Bayesian Framework, Feature Subspaces, and Progressive Learning

Relevance feedback: A power tool for interactive content-based image retrieval.

Relevance Feedback Learning with Feature Selection in Region-Based Image Retrieval

Similarity Measure Learning for Image Retrieval Using Feature Subspace Analysis

Evaluation of relevance feedback methods for 3D model retrieval