Abstract:Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these markets, which ultimately reduces workers' hourly wages. Invisible labor includes finding good tasks, messaging requesters, or managing payments. However, we currently know little about how much time crowd workers actually spend on invisible labor or how much it costs them economically. To ensure a fair and equitable future for crowd work, we need to be certain that workers are being paid fairly for all of the work they do. In this paper, we conduct a field study to quantify the invisible labor in crowd work. We build a plugin to record the amount of time that 100 workers on Amazon Mechanical Turk dedicate to invisible labor while completing 40,903 tasks. If we ignore the time workers spent on invisible labor, workers' median hourly wage was $3.76. But, we estimated that crowd workers in our study spent 33% of their time daily on invisible labor, dropping their median hourly wage to $2.83. We found that the invisible labor differentially impacts workers depending on their skill level and workers' demographics. The invisible labor category that took the most time and that was also the most common revolved around workers having to manage their payments. The second most time-consuming invisible labor category involved hyper-vigilance, where workers vigilantly watched over requesters' profiles for newly posted work or vigilantly searched for labor. We hope that through our paper, the invisible labor in crowdsourcing becomes more visible, and our results help to reveal the larger implications of the continuing invisibility of labor in crowdsourcing.

Undue Influence or Exploitation — A Qualitative Inquiry into an Ethical Dilemma Around Payment in Crowd Work-Based Research in the U.S.

What Scholars and IRBs Talk when They Talk about the Belmont Principles in Crowd Work-Based Research

The original sin of crowd work for human subjects research

Tragedy of the Commons in Crowd Work-Based Research

Tragedy of the Commons - A Critical Study of Data Quality and Validity Issues in Crowd Work-Based Research

Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing

Turning a Blind Eye to Team Members’ Unethical Behavior: the Role of Reward Systems

When the Automated Fire Backfires: the Adoption of Algorithm-based HR Decision-making Could Induce Consumer's Unfavorable Ethicality Inferences of the Company

Quality-Aware Incentive Mechanisms Under Social Influences in Data Crowdsourcing

Resolving the Human Subjects Status of Machine Learning's Crowdworkers

Two Ethical Dilemmas in Conducting Human Subjects Research on Amazon Mechanical Turk (Mturk)

A Collective Action Inspired Motivation Framework for Crowdsourcing

The Nature of Crowd Work and its Effects on Individuals’ Work Perception

Incentive Mechanism Design to Meet Task Criteria in Crowdsourcing: How to Determine Your Budget

Privacy in Crowdsourcing: a Review of the Threats and Challenges

Quantifying the Invisible Labor in Crowd Work

Fairness Preferences, Actual and Hypothetical: A Study of Crowdworker Incentives

The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices

Reputation-based Incentive Protocols in Crowdsourcing Applications

Medical crowdfunding in China: empirics and ethics

Online Decision Making in Crowdsourcing Markets: Theoretical Challenges (Position Paper)