Abstract:The design of a Web search evaluation metric is closely related with how the user's interaction process is modeled. Each behavioral model results in a different metric used to evaluate search performance. In these models and the user behavior assumptions behind them, when a user ends a search session is one of the prime concerns because it is highly related to both benefit and cost estimation. Existing metric design usually adopts some simplified criteria to decide the stopping time point: (1) upper limit for benefit (e.g. RR, AP); (2) upper limit for cost (e.g. Precision@N, DCG@N). However, in many practical search sessions (e.g. exploratory search), the stopping criterion is more complex than the simplified case. Analyzing benefit and cost of actual users' search sessions, we find that the stopping criteria vary with search tasks and are usually combination effects of both benefit and cost factors. Inspired by a popular computer game named Bejeweled, we propose a Bejeweled Player Model (BPM) to simulate users' search interaction processes and evaluate their search performances. In the BPM, a user stops when he/she either has found sufficient useful information or has no more patience to continue. Given this assumption, a new evaluation framework based on upper limits (either fixed or changeable as search proceeds) for both benefit and cost is proposed. We show how to derive a new metric from the framework and demonstrate that it can be adopted to revise traditional metrics like Discounted Cumulative Gain (DCG), Expected Reciprocal Rank (ERR) and Average Precision (AP). To show effectiveness of the proposed framework, we compare it with a number of existing metrics in terms of correlation between user satisfaction and the metrics based on a dataset that collects users' explicit satisfaction feedbacks and assessors' relevance judgements. Experiment results show that the framework is better correlated with user satisfaction feedbacks.

An Model of Website Traffic Estimation Based on Search Engines

An Evaluation Model on Information Coverage of Search Engines

A Model for Highway Traffic Noise Prediction Based on the Concept of Equivalent Traffic Flow

The Effects of Web Marketing for Chinese Travel Agencies with Content Analysis:Case Study of 44 Websites of Chinese Travel Agencies

Empirical Study of Keywords Biding Strategy and Search Engine Advertising Performance

Quantitative Index Construction of Website Influence: Taking China Private Universities as Example

Survey and analysis on the Internet traffic model

Traffic State and Emission Estimation for Urban Expressways Based on Heterogeneous Data

Empirical Analysis on Academic Journal Website Optimization Based on SEO

Dynamic Analysis of Chinese Search Engine User Behavior

A Study on Web Searching

Automatic Search Engine Performance Evaluation with Click-Through Data Analysis

A COMPARATIVE STUDY ON TRAFFIC FLOW MODELS BASED ON MEASURED DATA

Spatial Distribution and Difference Research of Chinese Urban Planning Websites

Evaluating Web Search with a Bejeweled Player Model.

A Factor Marginal Effect Analysis Approach and Its Application in E-Commerce Search System

Comparative study on Chinese meta-search engine and foreign meta-search engine

Automatic Search Engine Evaluation Based On User Behavior Analysis

Traffic Measurement Model Based on Traffic Matrix Estimation

Progress Of Attraction Websites In Mainland China: From 2005 To 2010

A novel multi-modal analysis model with Baidu Search Index for subway passenger flow forecasting