Abstract:Data annotation remains the sine qua non of machine learning and AI. Recent empirical work on data annotation has begun to highlight the importance of rater diversity for fairness, model performance, and new lines of research have begun to examine the working conditions for data annotation workers, the impacts and role of annotator subjectivity on labels, and the potential psychological harms from aspects of annotation work. This paper outlines a critical genealogy of data annotation; starting with its psychological and perceptual aspects. We draw on similarities with critiques of the rise of computerized lab-based psychological experiments in the 1970's which question whether these experiments permit the generalization of results beyond the laboratory settings within which these results are typically obtained. Do data annotations permit the generalization of results beyond the settings, or locations, in which they were obtained? Psychology is overly reliant on participants from Western, Educated, Industrialized, Rich, and Democratic societies (WEIRD). Many of the people who work as data annotation platform workers, however, are not from WEIRD countries; most data annotation workers are based in Global South countries. Social categorizations and classifications from WEIRD countries are imposed on non-WEIRD annotators through instructions and tasks, and through them, on data, which is then used to train or evaluate AI models in WEIRD countries. We synthesize evidence from several recent lines of research and argue that data annotation is a form of automated social categorization that risks entrenching outdated and static social categories that are in reality dynamic and changing. We propose a framework for understanding the interplay of the global social conditions of data annotation with the subjective phenomenological experience of data annotation work.

Wisdom for the Crowd: Discoursive Power in Annotation Instructions for Computer Vision

Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision

The Data-Production Dispositif

How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets

Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotation

CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation

Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy

Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation

The Problems with Proxies: Making Data Work Visible through Requester Practices

The Coloniality of Data Work in Latin America

Whose AI Dream? In search of the aspiration in data annotation

Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions

Incorporating Worker Perspectives into MTurk Annotation Practices for NLP

Challenges and Considerations in Annotating Legal Data: A Comprehensive Overview

Re-examining Sexism and Misogyny Classification with Annotator Attitudes

Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design

Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks

Unpacking Invisible Work Practices, Constraints, and Latent Power Relationships in Child Welfare through Casenote Analysis

In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers

Crowdsourcing in Computer Vision

Augmented Behavioral Annotation Tools, with Application to Multimodal Datasets and Models: A Systematic Review