Tragedy of the Commons - A Critical Study of Data Quality and Validity Issues in Crowd Work-Based Research

Huichuan Xia
DOI: https://doi.org/10.24251/hicss.2022.548
2022-01-01
Abstract:Academic scholars have leveraged crowd work platforms such as MTurk to conduct research and collect data.Though prior studies have discussed data quality and validity issues in crowd work via surveys and experiments, they kind of neglected to explore the scholars' and particularly the IRB's ethical concerns in these respects.In this study, we interviewed 17 scholars from six disciplines and 15 IRB directors and analysts in the U.S. to fill this research gap.We identified common themes among our respondents but also discovered distinctive and even opposing views regarding the approval rate, rejection, internal and external research validity.Based on the findings, we discussed a potential Tragedy of the Commons regarding the data quality deterioration and the disciplinary differences regarding validity in crowd work-based research.Finally, we advocated that the IRB's ethical concerns in crowd work-based research should be heard and respected further.
What problem does this paper attempt to address?