Abstract:Our objective is to estimate the relevance of an image to a query for image search purposes. We address two limitations of the existing image search engines in this paper. First, there is no straightforward way of bridging the gap between semantic textual queries as well as users' search intents and image visual content. Image search engines therefore primarily rely on static and textual features. Visual features are mainly used to identify potentially useful recurrent patterns or relevant training examples for complementing search by image reranking. Second, image rankers are trained on query-image pairs labeled by human experts, making the annotation intellectually expensive and time-consuming. Furthermore, the labels may be subjective when the queries are ambiguous, resulting in difficulty in predicting the search intention. We demonstrate that the aforementioned two problems can be mitigated by exploring the use of click-through data, which can be viewed as the footprints of user searching behavior, as an effective means of understanding query. The correspondences between an image and a query are determined by whether the image was searched and clicked by users under the query in a commercial image search engine. We therefore hypothesize that the image click counts in response to a query are as their relevance indications. For each new image, our proposed graph-based label propagation algorithm employs neighborhood graph search to find the nearest neighbors on an image similarity graph built up with visual representations from deep neural networks and further aggregates their clicked queries/click counts to get the labels of the new image. We conduct experiments on MSR-Bing Grand Challenge and the results show consistent performance gain over various baselines. In addition, the proposed approach is very efficient, completing annotation of each query-image pair within just 15 milliseconds on a regular PC.

Towards msr-bing challenge: Ensemble of diverse models for image retrieval

Image Retrieval by Cross-Media Relevance Fusion.

Cross-media relevance mining for evaluating text-based image search engine

Learning with Both Unlabeled Data and Query Logs for Image Search.

Improving Biomedical Named Entity Recognition with a Unified Multi-Task MRC Framework

Domain-Specific Modeling and Semantic Alignment for Image-Based 3d Model Retrieval

Bag-Of-Words Based Deep Neural Network For Image Retrieval

USTC-CityU at MSR-Bing IRC: Image search by graph-based label propagation

Flickr30K-CFQ: A Compact and Fragmented Query Dataset for Text-image Retrieval

Discriminant analysis of various concentric needle EMG and macro-EMG parameters in detecting myopathic abnormality

Transcending Fusion: A Multiscale Alignment Method for Remote Sensing Image–Text Retrieval

Msr asia msm at activitynet challenge 2016

Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models

Transcending Fusion: A Multi-Scale Alignment Method for Remote Sensing Image-Text Retrieval

MIBench: Evaluating Multimodal Large Language Models over Multiple Images

MNER-QG: An End-to-End MRC Framework for Multimodal Named Entity Recognition with Query Grounding

Multi-Grained Attention Network with Mutual Exclusion for Composed Query-Based Image Retrieval

MSRA-USTC-SJTU at TRECVID 2007: High-Level Feature Extraction and Search.

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

Unsupervised Collaborative Metric Learning with Mixed-Scale Groups for General Object Retrieval

Image search by graph-based label propagation with image representation from DNN.