Abstract:The design of a Web search evaluation metric is closely related with how the user's interaction process is modeled. Each behavioral model results in a different metric used to evaluate search performance. In these models and the user behavior assumptions behind them, when a user ends a search session is one of the prime concerns because it is highly related to both benefit and cost estimation. Existing metric design usually adopts some simplified criteria to decide the stopping time point: (1) upper limit for benefit (e.g. RR, AP); (2) upper limit for cost (e.g. Precision@N, DCG@N). However, in many practical search sessions (e.g. exploratory search), the stopping criterion is more complex than the simplified case. Analyzing benefit and cost of actual users' search sessions, we find that the stopping criteria vary with search tasks and are usually combination effects of both benefit and cost factors. Inspired by a popular computer game named Bejeweled, we propose a Bejeweled Player Model (BPM) to simulate users' search interaction processes and evaluate their search performances. In the BPM, a user stops when he/she either has found sufficient useful information or has no more patience to continue. Given this assumption, a new evaluation framework based on upper limits (either fixed or changeable as search proceeds) for both benefit and cost is proposed. We show how to derive a new metric from the framework and demonstrate that it can be adopted to revise traditional metrics like Discounted Cumulative Gain (DCG), Expected Reciprocal Rank (ERR) and Average Precision (AP). To show effectiveness of the proposed framework, we compare it with a number of existing metrics in terms of correlation between user satisfaction and the metrics based on a dataset that collects users' explicit satisfaction feedbacks and assessors' relevance judgements. Experiment results show that the framework is better correlated with user satisfaction feedbacks.

Research on Webpage Interest Rate Metrics and Its Application in Interest Model

Research on User Interest Model Optimization Introducing Drift Characteristics

A Content-Based Intelligent Ranking Model for Microblog

Dynamic Semantic Clustering Approach For Web User Interest

Modeling Lifetime of Web Pages Based on User Interest Analysis

Research on Personalized Search User Interest Updating Learning and Evaluation

A Web Recommendation Method Based on User Interest Model

A Intelligent Method of Modelling Web User Interest

Research on User Profile Based on Time Element Interest Rate Quantification Method and VSM Extension

Extraction Scheme of Theme Interest Rate and It's Application in User Profile

Evaluating Web Search with a Bejeweled Player Model.

Consumption Pricing Mechanism of Scientific and Technological Resources Based on Multi-Agent Game Theory: an Interactive Analytical Model and Experimental Validation

User Interest Mining Based on the Bayesian Probability

Impression space model for the evaluation of Internet advertising effectiveness

Research on Bayesian Network Based User’s Interest Model

Implicit Rating - A Case Study

RSS Item Ranking Based on Implicit Feedback

History Based User Interest Modeling In Www Access

Research on Decisive Mechanism of Internet Financial Interest Rate.

An Adaptive User Profile Based on Memory Model

User Interest Profile Identification Using Wikipedia Knowledge Database