Abstract:Test collections are information retrieval tools that allow researchers to quickly and easily evaluate ranking algorithms. While test collections have become an integral part of IR research, the process of data creation involves significant efforts in manual annotations, which often makes it very expensive and time-consuming. Thus, the test collections could become small when the budget is limited, which may lead to unstable evaluations. As an alternative, recent studies have proposed the use of large language models (LLMs) to completely replace human assessors. However, while LLMs seem to somewhat correlate with human judgments, they are not perfect and often show bias. Moreover, even if a well-performing LLM or prompt is found on one dataset, there is no guarantee that it will perform similarly in practice, due to difference in tasks and data. Thus a complete replacement with LLMs is argued to be too risky and not fully trustable. Thus, in this paper, we propose \textbf{L}LM-\textbf{A}ssisted \textbf{R}elevance \textbf{A}ssessments (\textbf{LARA}), an effective method to balance manual annotations with LLM annotations, which helps to make a rich and reliable test collection. We use the LLM's predicted relevance probabilities in order to select the most profitable documents to manually annotate under a budget constraint. While solely relying on LLM's predicted probabilities to manually annotate performs fairly well, with theoretical reasoning, LARA guides the human annotation process even more effectively via online calibration learning. Then, using the calibration model learned from the limited manual annotations, LARA debiases the LLM predictions to annotate the remaining non-assessed data. Empirical evaluations on TREC-COVID and TREC-8 Ad Hoc datasets show that LARA outperforms the alternative solutions under almost any budget constraint.

SARA: A Collection of Sensitivity-Aware Relevance Assessments

ACORDAR: A Test Collection for Ad Hoc Content-Based (RDF) Dataset Retrieval

LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?

Cheap IR Evaluation: Fewer Topics, No Relevance Judgements, and Crowdsourced Assessments

ACORDAR

From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Search

HC4: A New Suite of Test Collections for Ad Hoc CLIR

ACORDAR 2.0: A Test Collection for Ad Hoc Dataset Retrieval with Densely Pooled Datasets and Question-Style Queries

Understanding Relevance Judgments in Legal Case Retrieval

Synthetic Test Collections for Retrieval Evaluation

Understanding and Predicting Characteristics of Test Collections in Information Retrieval

Commonsense Knowledge Salience Evaluation with a Benchmark Dataset in E-commerce

Sensitive Content Classification in Social Media: A Holistic Resource and Evaluation

A Sensitivity Analysis of the MSMARCO Passage Collection

LeCaRD: A Legal Case Retrieval Dataset for Chinese Law System

Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably

A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look

ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures

Chinese Web Retrieval Test Collections: Construction, Analysis and Application

Sogou-QCL

SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval