Abstract:The design of a Web search evaluation metric is closely related with how the user's interaction process is modeled. Each behavioral model results in a different metric used to evaluate search performance. In these models and the user behavior assumptions behind them, when a user ends a search session is one of the prime concerns because it is highly related to both benefit and cost estimation. Existing metric design usually adopts some simplified criteria to decide the stopping time point: (1) upper limit for benefit (e.g. RR, AP); (2) upper limit for cost (e.g. Precision@N, DCG@N). However, in many practical search sessions (e.g. exploratory search), the stopping criterion is more complex than the simplified case. Analyzing benefit and cost of actual users' search sessions, we find that the stopping criteria vary with search tasks and are usually combination effects of both benefit and cost factors. Inspired by a popular computer game named Bejeweled, we propose a Bejeweled Player Model (BPM) to simulate users' search interaction processes and evaluate their search performances. In the BPM, a user stops when he/she either has found sufficient useful information or has no more patience to continue. Given this assumption, a new evaluation framework based on upper limits (either fixed or changeable as search proceeds) for both benefit and cost is proposed. We show how to derive a new metric from the framework and demonstrate that it can be adopted to revise traditional metrics like Discounted Cumulative Gain (DCG), Expected Reciprocal Rank (ERR) and Average Precision (AP). To show effectiveness of the proposed framework, we compare it with a number of existing metrics in terms of correlation between user satisfaction and the metrics based on a dataset that collects users' explicit satisfaction feedbacks and assessors' relevance judgements. Experiment results show that the framework is better correlated with user satisfaction feedbacks.

Integrating expert knowledge and multilingual web crawling data in a lead qualification system

Catching potential customers: an example of Web-mining-aided e-commerce decision-making

A Multi-View Semi-Supervised Approach for Task-Level Web Search Success Evaluation.

LeadsRobot: A Sales Leads Generation Robot Based on Big Data Analytics.

Intelligent User Assistance for Automated Data Mining Method Selection

Scalable and Noise Tolerant Web Knowledge Extraction for Search Task Simplification.

QAnswer: Towards Question Answering Search over Websites

Using supervised machine learning for B2B sales forecasting: A case study of spare parts sales forecasting at an after-sales service provider

Discovering development of personnel qualification requirements by web mining: a case study.

Decision Modeling and Evaluation of Enterprise Digital Transformation Using Data Mining

NLP-based Decision Support System for Examination of Eligibility Criteria from Securities Prospectuses at the German Central Bank

Integrating Kano Model, AHP, and QFD Methods for New Product Development Based on Text Mining, Intuitionistic Fuzzy Sets, and Customers Satisfaction

Evaluating Digital Library Search Systems by using Formal Process Modelling

Integrating deep learning and multi-attention for joint extraction of entities and relationships in engineering consulting texts

Evaluating Web Search with a Bejeweled Player Model.

Research on Automated Web Navigation and Data Integration Rules for Web Infor-mation Extraction

Knowledge Extraction from the Language Extended Lexicon Glossary Using Natural Language Processing

The Impact of Technical Domain Expertise on Search Behavior and Task Outcome

Connecting firm's web scraped textual content to body of science: Utilizing microsoft academic graph hierarchical topic modeling

Improving Company Valuations with Automated Knowledge Discovery, Extraction and Fusion

Dynamic Updating of the Knowledge Base for a Large-Scale Question Answering System