Abstract:The traditional evaluation of information retrieval (IR) systems is generally very costly as it requires manual relevance annotation from human experts. Recent advancements in generative artificial intelligence -- specifically large language models (LLMs) -- can generate relevance annotations at an enormous scale with relatively small computational costs. Potentially, this could alleviate the costs traditionally associated with IR evaluation and make it applicable to numerous low-resource applications. However, generated relevance annotations are not immune to (systematic) errors, and as a result, directly using them for evaluation produces unreliable results. In this work, we propose two methods based on prediction-powered inference and conformal risk control that utilize computer-generated relevance annotations to place reliable confidence intervals (CIs) around IR evaluation metrics. Our proposed methods require a small number of reliable annotations from which the methods can statistically analyze the errors in the generated annotations. Using this information, we can place CIs around evaluation metrics with strong theoretical guarantees. Unlike existing approaches, our conformal risk control method is specifically designed for ranking metrics and can vary its CIs per query and document. Our experimental results show that our CIs accurately capture both the variance and bias in evaluation based on LLM annotations, better than the typical empirical bootstrapping estimates. We hope our contributions bring reliable evaluation to the many IR applications where this was traditionally infeasible.

A review of relevance feedback experiments at the 2003 reliable information access (RIA) workshop.

Revisiting Iterative Relevance Feedback for Document and Passage Retrieval

A Document Relevance Based Search Result Re-Ranking

Relevance Feedback by Exploring the Different Feedback Sources and Collection Structure.

Implications of Inter-Rater Agreement on a Student Information Retrieval Evaluation

When does Relevance Mean Usefulness and User Satisfaction in Web Search?

Pseudo-Relevance Feedback Based On Mrmr Criteria

Expansiion-Based Technologies in Finding Relevant and New Information: THU TREC2002 Novelty Track Experiments

An Exam-based Evaluation Approach Beyond Traditional Relevance Judgments

Learning-based Relevance Feedback for Web-Based Relation Completion

Relevance Feedback Techniques In Interactive Content-Based Image Retrieval

Relevance Feedback: a Power Tool for Interactive Content-Based Image Retrieval

A Relevance Feedback Architecture for Content-Based Multimedia Information Retrieval Systems

Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I

Research of Relevance Feedback Technology in Personalized Information Retrieval

On Relevance in Information Retrieval Based on Relevance Theory

A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look

A Linguistic Study on Relevance Modeling in Information Retrieval

Usefulness as the Criterion for Evaluation of Interactive Information Retrieval

THUIR at TREC 2008: Relevance Feedback Track

Review on Relevance Criteria(2000-2010)