Abstract:Following the success of Cranfield-like evaluation approaches to evaluation in web search, web image search has also been evaluated with absolute judgments of (graded) relevance. However, recent research has found that collecting absolute relevance judgments may be difficult in image search scenarios due to the multi-dimensional nature of relevance for image results. Moreover, existing evaluation metrics based on absolute relevance judgments do not correlate well with search users' satisfaction perceptions in web image search. Unlike absolute relevance judgments, preference judgments do not require that relevance grades be pre-defined, i.e., how many levels to use and what those levels mean. Instead of considering each document in isolation, preference judgments consider a pair of documents and require judges to state their relative preference. Such preference judgments are usually more reliable than absolute judgments since the presence of (at least) two items establishes a certain context. While preference judgments have been studied extensively for general web search, there exists no thorough investigation on how preference judgments and preference-based evaluation metrics can be used to evaluate web image search systems. Compared to general web search, web image search may be an even better fit for preference-based evaluation because of its grid-based presentation style. The limited need for fresh results in web image search also makes preference judgments more reusable than for general web search. In this paper, we provide a thorough comparison of variants of preference judgments for web image search. We find that compared to strict preference judgments, weak preference judgments require less time and have better inter-assessor agreement. We also study how absolute relevance levels of two given images affect preference judgments between them. Furthermore, we propose a preference-based evaluation metric named Preference-Winning-Penalty (PWP) to evaluate and compare between two different image search systems. The proposed PWP metric outperforms existing evaluation metrics based on absolute relevance judgments in terms of agreement to system-level preferences of actual users.

Exploration of Image Search Results Quality Assessment

Learning to judge image search results.

Assessing Image Retrieval Quality at the First Glance

Progressive Image Retrieval with Quality Guarantee under MapReduce Framework

Improve Ranking by Using Image Information

An Adaptive Query Prototype Modeling Method for Image Search Reranking

From Document to Image: Learning a Scalable Ranking Model for Content Based Image Retrieval

Learning to Combine Ad-hoc Ranking Functions for Image Retrieval

Improving Object Retrieval Quality by Integration of Similarity Propagation and Query Expansion

Query Difficulty Prediction for Web Image Search

Query-Adaptive Image Search Re-Ranking Using Deep Convolutional Neural Network Feature

Preference-based Evaluation Metrics for Web Image Search

A comprehensive study on learning to rank for content-based image retrieval

Social-Sensed Image Search.

Quality Assessment on User Generated Image for Mobile Search Application.

A Novel and Adaptive Method for Image Search Reranking

Image Search Re-Ranking with an Improved VisualRank Algorithm and Multi-Layer DCNN Features.

Oracle in Image Search

An Effective and Efficient Re-ranking Framework for Social Image Search

The Role of Attractiveness in Web Image Search

Image Search Reranking with Query-Dependent Click-Based Relevance Feedback.