Abstract:Evaluation plays an essential way in Information Retrieval (IR) researches. Existing Web search evaluation methodologies usually come in two ways: offline and online methods. The benchmarks generated by offline methods (e.g. Cranfield-like ones) could be easily reused. However, the evaluation metrics in these methods are usually based on various user behavior assumptions (e.g. Cascade assumption) and may not well accord with actual user behaviors. Online methods, in contrast, can well capture users’ actual preferences while the results are not usually reusable. In this paper, we focus on the evaluation problem where users are using search engines to finish complex tasks. These tasks usually involve multiple queries in a single search session and propose challenges to both offline and online evaluation methodologies. To tackle this problem, we propose a search success evaluation framework based on machine translation model. In this framework, we formulate the search success evaluation problem as a machine translation evaluation problem: the ideal search outcome (i.e. necessary information to finish the task) is considered as the reference while search outcome from individual users (i.e. content that are perceived by users) as the translation. Thus, we adopt BLEU, a long standing machine translation evaluation metric, to evaluate the success of searchers. This framework avoids the introduction of possibly unreliable behavior assumptions and is reusable as well. We also tried a number of automatic methods which aim to minimize assessors’ efforts based on search interaction behavior such as eye-tracking and click-through. Experimental results indicate that the proposed evaluation method well correlates with explicit feedback on search satisfaction from search users. It is also suitable for search success evaluation when there is need for quick or frequent evaluations.

A Discussion on the Evaluation of Cross-Language Information Retrieval

Research on Chinese-English Cross-Language Information Retrieval

Domain-specific Cross-Language Relevant Question Retrieval.

Research On English-Chinese Bi-Directional Cross-Language Information Retrieval

Research on Lucene-based English-Chinese Cross-Language Information Retrieval

Review of Web Information Resources Evaluation

English-Chinese Cross-language Information Retrieval Using Lucene System

CUHK at ImageCLEF 2005: cross-language and cross-media image retrieval

A Review of Advanced Topics in Information Retrieval

Study on Cross Platform of Teaching Evaluation System

An Evaluation System for Algorithms of Automated Text Categorization on Chinese

Cross-Language Information Retrieval Based on Multiple Information

Dictionary-based Method for English-Chinese Bidirectional Cross-Language Information Retrieval

Using Context to Improve the Evaluation of Information Retrieval Systems

A re-examination of IR techniques in QA system

Discussion about Electronic Resources Construction and Evaluation Index

Evaluate Retrieval Systems Based on Ontology Vocabulary.

Technology Report of HIT-IRLab for Evaluation 2005 of 863 Information Retrieval

An Ontology-Based Cross Language Information Retrieval Model

Search Success Evaluation with Translation Model

Chinese-English Cross-Lingual Information Retrieval Based On Domain Ontology Knowledge