Abstract:Mechanical Turk ( MT urk), an online labor market run by Amazon, provides a web platform for conducting behavioral experiments; the site offers immediate and inexpensive access to a large subject pool. In this study, we review recent research about using MT urk for behavioral experiments and test the validity of using MT urk for experiments in behavioral operations management. We recruited subjects from MT urk to replicate the inventory management experiment from Bolton and Katok ( ), as well as the procurement auction experiment from Engelbrecht‐Wiggans and Katok ( ), and the supply chain contracting experiment from Loch and Wu ( ). We successfully replicate individual biases in the inventory management and procurement auction experiments, but learning in the individual tasks occurs more slowly on MT urk compared to the original studies. Further, we find that social preference manipulations in the supply chain experiment are ineffective in changing the behavior of MT urk subjects, in contrast to the original study. We conducted an additional replication study of the supply chain contracting experiment using student subjects in a standard laboratory. Results from this laboratory replication also fail to replicate the original laboratory study, indicating that the effect of social preferences on supply chain contracting may not be robust to alternative subject pools. We conclude that factors potentially influencing the differences observed on MT urk are less related to the online environment, but more related to the diversity and characteristics of subject pool on MT urk. Overall, MT urk appears to be an important and relevant tool for researchers in behavioral operations, but we caution researchers about slower learning of the MT urk subjects and the use of social preference manipulations on MT urk.

Examining the replicability of online experiments selected by a decision market

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Predicting replicability -- analysis of survey and prediction market data from large-scale forecasting projects

The Online Laboratory: Conducting Experiments in a Real Labor Market

Data from a Pre-Publication Independent Replication Initiative Examining Ten Moral Judgement Effects

Replicability of simulation studies for the investigation of statistical methods: the RepliSims project

Estimating the reproducibility of psychological science

Running Behavioral Operations Experiments Using Amazon's Mechanical Turk

Replicating patterns of prospect theory for decision under risk

Can laypeople predict the replicability of social science studies without expert intervention: an exploratory study

A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Revisiting representativeness heuristic classic paradigms: Replication and extensions of nine experiments in Kahneman and Tversky (1972)

Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

The Pipeline Project: Pre-Publication Independent Replications of A Single Laboratory'S Research Pipeline

Having less, giving more? Two preregistered replications of the relationship between social class and prosocial behavior

Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

Crowdsourcing Hypothesis Tests: Making Transparent How Design Choices Shape Research Results.

Can we trust online crowdworkers? Comparing online and offline participants in a preference test of virtual agents

A large-scale in silico replication of ecological and evolutionary studies

Replication: Revisiting Tversky and Shafir’s (1992) Disjunction Effect with an extension comparing between and within subject designs

The power of social influence: A replication and extension of the Asch experiment