Abstract:The design of a Web search evaluation metric is closely related with how the user's interaction process is modeled. Each behavioral model results in a different metric used to evaluate search performance. In these models and the user behavior assumptions behind them, when a user ends a search session is one of the prime concerns because it is highly related to both benefit and cost estimation. Existing metric design usually adopts some simplified criteria to decide the stopping time point: (1) upper limit for benefit (e.g. RR, AP); (2) upper limit for cost (e.g. Precision@N, DCG@N). However, in many practical search sessions (e.g. exploratory search), the stopping criterion is more complex than the simplified case. Analyzing benefit and cost of actual users' search sessions, we find that the stopping criteria vary with search tasks and are usually combination effects of both benefit and cost factors. Inspired by a popular computer game named Bejeweled, we propose a Bejeweled Player Model (BPM) to simulate users' search interaction processes and evaluate their search performances. In the BPM, a user stops when he/she either has found sufficient useful information or has no more patience to continue. Given this assumption, a new evaluation framework based on upper limits (either fixed or changeable as search proceeds) for both benefit and cost is proposed. We show how to derive a new metric from the framework and demonstrate that it can be adopted to revise traditional metrics like Discounted Cumulative Gain (DCG), Expected Reciprocal Rank (ERR) and Average Precision (AP). To show effectiveness of the proposed framework, we compare it with a number of existing metrics in terms of correlation between user satisfaction and the metrics based on a dataset that collects users' explicit satisfaction feedbacks and assessors' relevance judgements. Experiment results show that the framework is better correlated with user satisfaction feedbacks.

A Fused Score for Ranking and Evaluation Using Multiple Performance Metrics

Ranking evaluation metrics from a group-theoretic perspective

Foundations of the Theory of Performance-Based Ranking

Integrating Fuzzy C-Means and TOPSIS for performance evaluation: An application and comparative analysis

Score Design for Multi-Criteria Incentivization

Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics

Multi-Attribute Comprehensive Evaluation Method for Power Transmission and Transformation Projects

Ranking Multi-Metric Scientific Achievements Using a Concept of Pareto Optimality

A Novel Score Function Determined by the Residual Sector Area on PFNs Space and Its Application in Fuzzy Decision-Making

A Strategy on Selecting Performance Metrics for Classifier Evaluation.

Fuzzy comprehensive performance evaluation method of rolling linear guide based on improved analytic hierarchy process

A comparative study of scoring systems by simulations

Corporate Social Responsibility Performance-Evaluation Based on Analytic Hierarchy Process-Fuzzy Comprehensive Evaluation Model

Metric-agnostic Ranking Optimization

Fuzzy Divergence Measure Based on Technique for Order of Preference by Similarity to Ideal Solution Method for Staff Performance Appraisal

Two-level multi-criteria comprehensive evaluation for preference vectors in online shopping platform evaluation

Research on Comprehensive Quality Evaluation Method Based on Cooperative Performance.

A New Weighted Spearman's Footrule as A Measure of Distance between Rankings

Evaluating Web Search with a Bejeweled Player Model.

Ranking Consistent Rate: New Evaluation Criterion on Pairwise Subjective Experiments