Abstract:Image reranking is effective for improving the performance of a text-based image search. However, existing reranking algorithms are limited for two main reasons: 1) the textual meta-data associated with images is often mismatched with their actual visual content and 2) the extracted visual features do not accurately describe the semantic similarities between images. Recently, user click information has been used in image reranking, because clicks have been shown to more accurately describe the relevance of retrieved images to search queries. However, a critical problem for click-based methods is the lack of click data, since only a small number of web images have actually been clicked on by users. Therefore, we aim to solve this problem by predicting image clicks. We propose a multimodal hypergraph learning-based sparse coding method for image click prediction, and apply the obtained click data to the reranking of images. We adopt a hypergraph to build a group of manifolds, which explore the complementarity of different features through a group of weights. Unlike a graph that has an edge between two vertices, a hyperedge in a hypergraph connects a set of vertices, and helps preserve the local smoothness of the constructed sparse codes. An alternating optimization procedure is then performed, and the weights of different modalities and the sparse codes are simultaneously obtained. Finally, a voting strategy is used to describe the predicted click as a binary event (click or no click), from the images' corresponding sparse codes. Thorough empirical studies on a large-scale database including nearly 330 K images demonstrate the effectiveness of our approach for click prediction when compared with several other methods. Additional image reranking experiments on real-world data show the use of click prediction is beneficial to improving the performance of prominent graph-based image reranking algorithms.

Multimodal graph-based reranking for web image search.

Optimizing Multimodal Reranking for Web Image Search

Contextual Modeling on Auxiliary Points for Robust Image Reranking

Visual Reranking through Weakly Supervised Multi-graph Learning

Image Search Reranking with Query-Dependent Click-Based Relevance Feedback.

Click-boosting multi-modality graph-based reranking for image search

Online Metric Learning for Relevance Feedback in E-Commerce Image Retrieval Applications

Weakly Supervised Multi-Graph Learning for Robust Image Reranking

Click prediction for web image reranking using multimodal sparse coding.

An Attribute-Assisted Reranking Model for Web Image Search.

Visual reranking with improved image graph

Multimodal Fusion for Video Search Reranking

Learning to rerank images with enhanced spatial verification

Heterogeneous Graph Propagation for Large-Scale Web Image Search.

Attribute-assisted Reranking for Web Image Retrieval

A Novel and Adaptive Method for Image Search Reranking

An Adaptive Query Prototype Modeling Method for Image Search Reranking

Multi-graph Enabled Active Learning for Multimodal Web Image Retrieval

Multimedia search reranking

Exploiting Click Constraints and Multi-view Features for Image Re-ranking

Effective Deep Learning-Based Multi-Modal Retrieval