Abstract:Evaluation plays an essential way in Information Retrieval (IR) researches. Existing Web search evaluation methodologies usually come in two ways: offline and online methods. The benchmarks generated by offline methods (e.g. Cranfield-like ones) could be easily reused. However, the evaluation metrics in these methods are usually based on various user behavior assumptions (e.g. Cascade assumption) and may not well accord with actual user behaviors. Online methods, in contrast, can well capture users’ actual preferences while the results are not usually reusable. In this paper, we focus on the evaluation problem where users are using search engines to finish complex tasks. These tasks usually involve multiple queries in a single search session and propose challenges to both offline and online evaluation methodologies. To tackle this problem, we propose a search success evaluation framework based on machine translation model. In this framework, we formulate the search success evaluation problem as a machine translation evaluation problem: the ideal search outcome (i.e. necessary information to finish the task) is considered as the reference while search outcome from individual users (i.e. content that are perceived by users) as the translation. Thus, we adopt BLEU, a long standing machine translation evaluation metric, to evaluate the success of searchers. This framework avoids the introduction of possibly unreliable behavior assumptions and is reusable as well. We also tried a number of automatic methods which aim to minimize assessors’ efforts based on search interaction behavior such as eye-tracking and click-through. Experimental results indicate that the proposed evaluation method well correlates with explicit feedback on search satisfaction from search users. It is also suitable for search success evaluation when there is need for quick or frequent evaluations.

"Satisfaction with Failure" or "Unsatisfied Success": Investigating the Relationship between Search Success and User Satisfaction.

Search Task Success Evaluation by Exploiting Multi-View Active Semi-Supervised Learning.

A Multi-View Semi-Supervised Approach for Task-Level Web Search Success Evaluation.

Satisfaction Prediction of Web Search Users

Search Action Sequence Modeling With Long Short-Term Memory For Search Task Success Evaluation

Investigating Cognitive Effects in Session-level Search User Satisfaction

Analysis into the Relationship Between Search Engine User Behavior and User Satisfaction Evaluation

Different Users, Different Opinions: Predicting Search Satisfaction With Mouse Movement Information

Are self-assessment of search ability and performance reliable?

Still Haven't Found What You're Looking For -- Detecting the Intent of Web Search Missions from User Interaction Features

Towards Designing Better Session Search Evaluation Metrics

User Intent, Behaviour, and Perceived Satisfaction in Product Search.

Towards Better Understanding of User Satisfaction in Open-Domain Conversational Search

Search Success Evaluation with Translation Model

Behavior-based evaluation of session satisfaction

User Satisfaction Prediction with Mouse Movement Information in Heterogeneous Search Environment

Investigation Of User Search Behavior While Facing Heterogeneous Search Services

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics

Understanding and Predicting Usefulness Judgment in Web Search

How do Information Source Selection Strategies Influence Users' Learning Outcomes'.