Abstract:The quality of datasets plays an increasingly crucial role in the research and development of modern artificial intelligence (AI). Despite the proliferation of open dataset platforms nowadays, data quality issues, such as insufficient documentation, inaccurate annotations, and ethical concerns, remain common in datasets widely used in AI. Furthermore, these issues are often subtle and difficult to be detected by rule-based scripts, requiring expensive manual identification and verification by dataset users or maintainers. With the increasing capability of large language models (LLMs), it is promising to streamline the curation of datasets with LLM agents. In this work, as the initial step towards this goal, we propose a dataset curation agent benchmark, DCA-Bench, to measure LLM agents' capability of detecting hidden dataset quality issues. Specifically, we collect diverse real-world dataset quality issues from eight open dataset platforms as a testbed. Additionally, to establish an automatic pipeline for evaluating the success of LLM agents, which requires a nuanced understanding of the agent outputs, we implement a dedicated Evaluator using another LLM agent. We demonstrate that the LLM-based Evaluator empirically aligns well with human evaluation, allowing reliable automatic evaluation on the proposed benchmark. We further conduct experiments on several baseline LLM agents on the proposed benchmark and demonstrate the complexity of the task, indicating that applying LLMs to real-world dataset curation still requires further in-depth exploration and innovation. Finally, the proposed benchmark can also serve as a testbed for measuring the capability of LLMs in problem discovery rather than just problem-solving. The benchmark suite is available at \url{<a class="link-external link-https" href="https://github.com/TRAIS-Lab/dca-bench" rel="external noopener nofollow">this https URL</a>}.

ACORDAR: A Test Collection for Ad Hoc Content-Based (RDF) Dataset Retrieval

ACORDAR

ACORDAR 2.0: A Test Collection for Ad Hoc Dataset Retrieval with Densely Pooled Datasets and Question-Style Queries

A LDA Topic Model Based Collection Selection Method for Distributed Information Retrieval

HC4: A New Suite of Test Collections for Ad Hoc CLIR

An Empirical Investigation of Implicit and Explicit Knowledge-Enhanced Methods for Ad Hoc Dataset Retrieval

Understanding and Predicting Characteristics of Test Collections in Information Retrieval

SARA: A Collection of Sensitivity-Aware Relevance Assessments

SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval

AToMiC: An Image/Text Retrieval Test Collection to Support Multimedia Content Creation

RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering

UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis

A Robust Computerized Adaptive Testing Approach in Educational Question Retrieval

ACR: A Benchmark for Automatic Cohort Retrieval

Enhancing Dataset Search with Compact Data Snippets

DCA-Bench: A Benchmark for Dataset Curation Agents

DAPR: A Benchmark on Document-Aware Passage Retrieval

Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration

AmazonQAC: A Large-Scale, Naturalistic Query Autocomplete Dataset

Synthetic Test Collections for Retrieval Evaluation

Dense Re-Ranking with Weak Supervision for RDF Dataset Search