Abstract:The global surge in AI applications is transforming industries, leading to displacement and complementation of existing jobs, while also giving rise to new employment opportunities. Data annotation, encompassing the labelling of images or annotating of texts by human workers, crucially influences the quality of a dataset directly influences the quality of AI models trained on it. This paper delves into the economics of data annotation, with a specific focus on the impact of task instruction design (that is, the choice between rules and standards as theorised in law and economics) and monetary incentives on data quality and costs. An experimental study involving 307 data annotators examines six groups with varying task instructions (norms) and monetary incentives. Results reveal that annotators provided with clear rules exhibit higher accuracy rates, outperforming those with vague standards by 14%. Similarly, annotators receiving an additional monetary incentive perform significantly better, with the highest accuracy rate recorded in the group working with both clear rules and incentives (87.5% accuracy). In addition, our results show that rules are perceived as being more helpful by annotators than standards and reduce annotators' difficulty in annotating images. These empirical findings underscore the double benefit of rule-based instructions on both data quality and worker wellbeing. Our research design allows us to reveal that, in our study, rules are more cost-efficient in increasing accuracy than monetary incentives. The paper contributes experimental insights to discussions on the economical, ethical, and legal considerations of AI technologies. Addressing policymakers and practitioners, we emphasise the need for a balanced approach in optimising data annotation processes for efficient and ethical AI development and usage.

Best Practices for Managing Data Annotation Projects

Survey of Data Annotation

Analyzing Dataset Annotation Quality Management in the Wild

Text Annotation Handbook: A Practical Guide for Machine Learning Projects

Human-centred Design on Crowdsourcing Annotation Towards Improving Active Learning Model Performance

Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy

Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets

Whose AI Dream? In search of the aspiration in data annotation

Challenges and Considerations in Annotating Legal Data: A Comprehensive Overview

anndata: Annotated data

A Qualitative Analysis of Common Practices in Annotations: A Taxonomy and Design Space

Best Practices for Text Annotation with Large Language Models

Managing Bias in Human-Annotated Data: Moving Beyond Bias Removal

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

Practical Annotation Strategies for Question Answering Datasets

Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks

Augmented Behavioral Annotation Tools, with Application to Multimodal Datasets and Models: A Systematic Review

Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Model

Situated Ground Truths: Enhancing Bias-Aware AI by Situating Data Labels with SituAnnotate

Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision

Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology