Abstract:Traditional information retrieval research has mostly focussed on satisfying clearly specified information needs. However, in reality, queries are often ambiguous and/or underspecified. In light of this, evaluating search result diversity is beginning to receive attention. We propose simple evaluation metrics for diversified Web search results. Our presumptions are that one or more interpretations (or intents) are possible for each given query, and that graded relevance assessments are available for intent-document pairs (as opposed to query-document pairs). Our goals are (a) to retrieve documents that cover as many intents as possible; and (b) to rank documents that are highly relevant to more popular intents higher than those that are marginally relevant to less popular intents. Unlike the Intent-Aware (IA) metrics proposed by Agrawal et al., our metrics successfully avoid ignoring minor intents. Unlike α-nDCG proposed by Clarke et al., our metrics can accomodate (i) which intents are more likely than others for a given query; and (ii) graded relevance within each intent. Furthermore, unlike these existing metrics, our metrics do not require approximation, and they range between 0 and 1. Experiments with the binary-relevance Diversity Task data from the TREC 2009 Web Track suggest that our metrics corrrelate well with existing metrics but can be more intuitive. Hence, we argue that our metrics are suitable for diversity evaluation given either the intent likelihood information or per-intent graded relevance, or preferably both.

Analysis of Methods for Novel Case Selection

Novelty and Diversity in Top-N Recommendation - Analysis and Evaluation

Research of new strategies for improving CBR system

A Hybrid Retrieval Strategy for Case-Based Reasoning Using Soft Likelihood Functions

A novel retrieval strategy for case-based reasoning based on attitudinal Choquet integral

A Kind of Case Similarity Evaluation Model Based on Case-Based Reasoning

ON CASE SIMILARITY MEASURES IN CBR

Design of Case-Based Reasoning System with Community Detection in Complexity Theory

Case Weighted Similarity Measure Optimization Method Based On Black Hole Cuckoo Search Algorithm

Research on Ontology-Based Case Indexing in CBR

Improving the Reliability of Case-Based Reasoning Systems

A Review and Analysis of Case-Based Reasoning Research

An Integrated Feature Selection and Cluster Analysis Techniques for Case-Based Reasoning

A Case Retrieval Strategy for Traffic Congestion Based on Cluster Analysis

Discussion on Some Problems about Assessing Case Similarity in Case-Based Reasoning Systems

An improved case-based reasoning method based on fuzzy clustering and mutual information

A Similarity Measure Based on Extension Distance and Its Application in CBR

Hybrid Similarity Measure for Case Retrieval in CBR and Its Application to Emergency Response Towards Gas Explosion

A Study on the Reliability of Case-Based Reasoning Systems

Simple Evaluation Metrics for Diversified Search Results.