Abstract:The design of a Web search evaluation metric is closely related with how the user's interaction process is modeled. Each behavioral model results in a different metric used to evaluate search performance. In these models and the user behavior assumptions behind them, when a user ends a search session is one of the prime concerns because it is highly related to both benefit and cost estimation. Existing metric design usually adopts some simplified criteria to decide the stopping time point: (1) upper limit for benefit (e.g. RR, AP); (2) upper limit for cost (e.g. Precision@N, DCG@N). However, in many practical search sessions (e.g. exploratory search), the stopping criterion is more complex than the simplified case. Analyzing benefit and cost of actual users' search sessions, we find that the stopping criteria vary with search tasks and are usually combination effects of both benefit and cost factors. Inspired by a popular computer game named Bejeweled, we propose a Bejeweled Player Model (BPM) to simulate users' search interaction processes and evaluate their search performances. In the BPM, a user stops when he/she either has found sufficient useful information or has no more patience to continue. Given this assumption, a new evaluation framework based on upper limits (either fixed or changeable as search proceeds) for both benefit and cost is proposed. We show how to derive a new metric from the framework and demonstrate that it can be adopted to revise traditional metrics like Discounted Cumulative Gain (DCG), Expected Reciprocal Rank (ERR) and Average Precision (AP). To show effectiveness of the proposed framework, we compare it with a number of existing metrics in terms of correlation between user satisfaction and the metrics based on a dataset that collects users' explicit satisfaction feedbacks and assessors' relevance judgements. Experiment results show that the framework is better correlated with user satisfaction feedbacks.

Interactive Approach to Eliciting Users' Preference Using Comparisons

An Approach to the Extraction of Preference-Related Information from Design Team Language

User Preference Quantity Versus Recommendation Performance: A Preliminary Study

A Probabilistic Approach for Extracting Design Preferences from Design Team Discussion

A Comparison of Formal Methods for Evaluating the Language of Preference in Engineering Design

SOCIAL INTEREST FOR USER SELECTING ITEMS IN RECOMMENDER SYSTEMS

A Model for Computing User's Preference Based on EP Algorithm

Adapting Ranking Functions to User Preference.

Facilitating Apps Recommendation in Google Play

Select-the-Best-Ones: A new way to judge relative relevance

Combining User Preferences and User Opinions for Accurate Recommendation

Learning Recency Based Comparative Choice Towards Point-of-interest Recommendation

Revisiting The Evaluation Of Diversified Search Evaluation Metrics With User Preferences

A textual data-driven method to identify and prioritise user preferences based on regret/rejoicing perception for smart and connected products

Robust consumer preference analysis with a social network

A Context-Aware Mobile User Behavior Based Preference Neighbor Finding Approach for Personalized Information Retrieval

Evaluating Web Search with a Bejeweled Player Model.

A Multi-Criteria Decision Support Model for Restaurant Selection Based on Users' Demand Level: The Case of Dianping.com

An Interactive Method To Solve The Priorities Of Attributes While The Preferences Of Evaluated Units Are Under Considering

Auto-generated Relative Importance for Multi-agent Inducing Variable in Uncertain and Preference Involved Evaluation