Abstract:Following the success of Cranfield-like evaluation approaches to evaluation in web search, web image search has also been evaluated with absolute judgments of (graded) relevance. However, recent research has found that collecting absolute relevance judgments may be difficult in image search scenarios due to the multi-dimensional nature of relevance for image results. Moreover, existing evaluation metrics based on absolute relevance judgments do not correlate well with search users' satisfaction perceptions in web image search. Unlike absolute relevance judgments, preference judgments do not require that relevance grades be pre-defined, i.e., how many levels to use and what those levels mean. Instead of considering each document in isolation, preference judgments consider a pair of documents and require judges to state their relative preference. Such preference judgments are usually more reliable than absolute judgments since the presence of (at least) two items establishes a certain context. While preference judgments have been studied extensively for general web search, there exists no thorough investigation on how preference judgments and preference-based evaluation metrics can be used to evaluate web image search systems. Compared to general web search, web image search may be an even better fit for preference-based evaluation because of its grid-based presentation style. The limited need for fresh results in web image search also makes preference judgments more reusable than for general web search. In this paper, we provide a thorough comparison of variants of preference judgments for web image search. We find that compared to strict preference judgments, weak preference judgments require less time and have better inter-assessor agreement. We also study how absolute relevance levels of two given images affect preference judgments between them. Furthermore, we propose a preference-based evaluation metric named Preference-Winning-Penalty (PWP) to evaluate and compare between two different image search systems. The proposed PWP metric outperforms existing evaluation metrics based on absolute relevance judgments in terms of agreement to system-level preferences of actual users.

Probabilistic Model-Based Relevance Evaluation of Famous People's Web Pages

A Probabilistic Method for Tag Ranking in Tagging System

Probabilistic factor models for web site recommendation.

A Model Based on Cocitation for Web Information Retrieval

A Relevance Evaluation Model of Information Retrieval Based on Weighted Term Frequency

Relevance Estimation with Multiple Information Sources on Search Engine Result Pages.

Probabilistic models for personalizing web search

A Document Relevance Based Search Result Re-Ranking

Predicting Relevance based on Assessor Disagreement: Analysis and Practical Applications for Search Evaluation

A Personalized Re-ranking Algorithm Based on Relevance Feedback

Preference-based Evaluation Metrics for Web Image Search

Celebrity Recommendation with Collaborative Social Topic Regression.

User-Oriented Relevance Judgment: A Conceptual Model

Ranking Websites: A Probabilistic View

XRank: Learning More fromWeb User Behaviors

Topic-Based Computing Model for Web Page Popularity and Website Influence

Select-the-Best-Ones: A new way to judge relative relevance

Understanding User Situational Relevance in Ranking Web Search Results

A Powerful Relevance Feedback Mechanism for Content-Based 3D Model Retrieval

Improving Web Search Ranking by Incorporating Summarization

Modeling Perceived Relevance for Tail Queries without Click-Through Data