Abstract:The design of a Web search evaluation metric is closely related with how the user's interaction process is modeled. Each behavioral model results in a different metric used to evaluate search performance. In these models and the user behavior assumptions behind them, when a user ends a search session is one of the prime concerns because it is highly related to both benefit and cost estimation. Existing metric design usually adopts some simplified criteria to decide the stopping time point: (1) upper limit for benefit (e.g. RR, AP); (2) upper limit for cost (e.g. Precision@N, DCG@N). However, in many practical search sessions (e.g. exploratory search), the stopping criterion is more complex than the simplified case. Analyzing benefit and cost of actual users' search sessions, we find that the stopping criteria vary with search tasks and are usually combination effects of both benefit and cost factors. Inspired by a popular computer game named Bejeweled, we propose a Bejeweled Player Model (BPM) to simulate users' search interaction processes and evaluate their search performances. In the BPM, a user stops when he/she either has found sufficient useful information or has no more patience to continue. Given this assumption, a new evaluation framework based on upper limits (either fixed or changeable as search proceeds) for both benefit and cost is proposed. We show how to derive a new metric from the framework and demonstrate that it can be adopted to revise traditional metrics like Discounted Cumulative Gain (DCG), Expected Reciprocal Rank (ERR) and Average Precision (AP). To show effectiveness of the proposed framework, we compare it with a number of existing metrics in terms of correlation between user satisfaction and the metrics based on a dataset that collects users' explicit satisfaction feedbacks and assessors' relevance judgements. Experiment results show that the framework is better correlated with user satisfaction feedbacks.

Cumulative Probability Distribution Model for Evaluating User Behavior Prediction Algorithms

User Preference Quantity Versus Recommendation Performance: A Preliminary Study

A Naive Bayesian Belief Network Model for Predicting Effort Deviation Rate in Software Testing

Label-free Evaluation for Performance of Fault Diagnosis Model on Unknown Distribution Dataset

Decision Curve Analysis: a Technical Note

Enhancing Collaborative Filtering Recommendation by User Interest Probability

A novel user behavior analysis and prediction algorithm based on mobile social environment

A new probabilistic model: Theory, simulation and applications to sports and failure times data

Product Adoption Rate Prediction in a Competitive Market

Evaluating Web Search with a Bejeweled Player Model.

Rating Prediction in Recommender Systems Based on User Behavior Probability and Complex Network Modeling

A Distribution-Free Method for Probabilistic Prediction

Product Adoption Rate Prediction: A Multi-factor View.

Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics

[Comparison Study of Model Evaluation Methods: Normalized Prediction Distribution Errors Vs. Visual Predictive Check].

Collaborative-Enhanced Prediction of Spending on Newly Downloaded Mobile Games under Consumption Uncertainty

Language Model Based Interactive Estimation of Distribution Algorithm

Online performance and proactive maintenance assessment of data driven prediction models

Avoiding C-hacking when evaluating survival distribution predictions with discrimination measures

Model‐based evaluation for online shopping platform with probabilistic double hierarchy linguistic CODAS method

Probability-Based Failure Evaluation for Power Measuring Equipment