Abstract:Mechanical Turk ( MT urk), an online labor market run by Amazon, provides a web platform for conducting behavioral experiments; the site offers immediate and inexpensive access to a large subject pool. In this study, we review recent research about using MT urk for behavioral experiments and test the validity of using MT urk for experiments in behavioral operations management. We recruited subjects from MT urk to replicate the inventory management experiment from Bolton and Katok ( ), as well as the procurement auction experiment from Engelbrecht‐Wiggans and Katok ( ), and the supply chain contracting experiment from Loch and Wu ( ). We successfully replicate individual biases in the inventory management and procurement auction experiments, but learning in the individual tasks occurs more slowly on MT urk compared to the original studies. Further, we find that social preference manipulations in the supply chain experiment are ineffective in changing the behavior of MT urk subjects, in contrast to the original study. We conducted an additional replication study of the supply chain contracting experiment using student subjects in a standard laboratory. Results from this laboratory replication also fail to replicate the original laboratory study, indicating that the effect of social preferences on supply chain contracting may not be robust to alternative subject pools. We conclude that factors potentially influencing the differences observed on MT urk are less related to the online environment, but more related to the diversity and characteristics of subject pool on MT urk. Overall, MT urk appears to be an important and relevant tool for researchers in behavioral operations, but we caution researchers about slower learning of the MT urk subjects and the use of social preference manipulations on MT urk.

An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use

Amazon's Mechanical Turk

Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations

Running experiments on Amazon Mechanical Turk

Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

Conducting behavioral research on Amazon’s Mechanical Turk

Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences

Comparing Amazon’s Mechanical Turk Platform to Conventional Data Collection Methods in the Health and Medical Research Literature

Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup

An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results

The Burden for High-Quality Online Data Collection Lies With Researchers, Not Recruitment Platforms

Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages

The MTurkification of Social and Personality Psychology

Running Behavioral Operations Experiments Using Amazon's Mechanical Turk

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

Beyond AMT: An Analysis of Crowd Work Platforms

Crowdsourcing Consumer Research

Lessons Learned from an Experiment in Crowdsourcing Complex Citizen Engineering Tasks with Amazon Mechanical Turk

Crowd Work on a CV? Understanding How AMT Fits into Turkers' Career Goals and Professional Profiles

Using Mechanical Turk to Build Machine Translation Evaluation Sets