Nonsensical results plague online social science tool

Cathleen O'Grady

DOI: https://doi.org/10.1126/science.adq4549

IF: 56.9

2024-05-18

Science

Abstract:Data quality suffers in some MTurk studies—but participant screening and other safeguards can help

multidisciplinary sciences

What problem does this paper attempt to address?

An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results

Michael Chmielewski,Sarah C. Kucker

DOI: https://doi.org/10.1177/1948550619875149

2019-10-10

Social Psychological and Personality Science

Abstract:Amazon’s Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns arose regarding MTurk data quality leading to questions about the utility of MTurk for psychological research. We present empirical evidence of a substantial decrease in data quality using a four-wave naturalistic experimental design: pre-, during, and post-summer 2018. During and to some extent post-summer 2018, we find significant increases in participants failing response validity indicators, decreases in reliability and validity of a widely used personality measure, and failures to replicate well-established findings. However, these detrimental effects can be mitigated by using response validity indicators and screening the data. We discuss implications and offer suggestions to ensure data quality.

psychology, social
The Burden for High-Quality Online Data Collection Lies With Researchers, Not Recruitment Platforms

Christine Cuskley,Justin Sulik

DOI: https://doi.org/10.1177/17456916241242734

Abstract:A recent article in Perspectives on Psychological Science (Webb & Tangney, 2022) reported a study in which just 2.6% of participants recruited on Amazon's Mechanical Turk (MTurk) were deemed "valid." The authors highlighted some well-established limitations of MTurk, but their central claims-that MTurk is "too good to be true" and that it captured "only 14 human beings . . . [out of] N = 529"-are radically misleading, yet have been repeated widely. This commentary aims to (a) correct the record (i.e., by showing that Webb and Tangney's approach to data collection led to unusually low data quality) and (b) offer a shift in perspective for running high-quality studies online. Negative attitudes toward MTurk sometimes reflect a fundamental misunderstanding of what the platform offers and how it should be used in research. Beyond pointing to research that details strategies for effective design and recruitment on MTurk, we stress that MTurk is not suitable for every study. Effective use requires specific expertise and design considerations. Like all tools used in research-from advanced hardware to specialist software-the tool itself places constraints on what one should use it for. Ultimately, high-quality data is the responsibility of the researcher, not the crowdsourcing platform.
Data quality of platforms and panels for online behavioral research

Peer Eyal,Rothschild David,Gordon Andrew,Evernden Zak,Damer Ekaterina

DOI: https://doi.org/10.3758/s13428-021-01694-3

IF: 5.953

2021-09-29

Behavior Research Methods

Abstract:We examine key aspects of data quality for online behavioral research between selected platforms (Amazon Mechanical Turk, CloudResearch, and Prolific) and panels (Qualtrics and Dynata). To identify the key aspects of data quality, we first engaged with the behavioral research community to discover which aspects are most critical to researchers and found that these include attention, comprehension, honesty, and reliability. We then explored differences in these data quality aspects in two studies (N ~ 4000), with or without data quality filters (approval ratings). We found considerable differences between the sites, especially in comprehension, attention, and dishonesty. In Study 1 (without filters), we found that only Prolific provided high data quality on all measures. In Study 2 (with filters), we found high data quality among CloudResearch and Prolific. MTurk showed alarmingly low data quality even with data quality filters. We also found that while reputation (approval rating) did not predict data quality, frequency and purpose of usage did, especially on MTurk: the lowest data quality came from MTurk participants who report using the site as their main source of income but spend few hours on it per week. We provide a framework for future investigation into the ever-changing nature of data quality in online research, and how the evolving set of platforms and panels performs on these key aspects.

psychology, experimental, mathematical
Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA

Benjamin D. Douglas,Patrick J. Ewell,Markus Brauer

DOI: https://doi.org/10.1371/journal.pone.0279720

IF: 3.7

2023-03-15

PLoS ONE

Abstract:With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher's question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ( 2.00) were cheaper than MTurk ( 8.17). SONA cost $0.00, yet took the longest to collect the data.

multidisciplinary sciences
Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire

Rachel Pozzar,Marilyn J Hammer,Meghan Underhill-Blazey,Alexi A Wright,James A Tulsky,Fangxin Hong,Daniel A Gundersen,Donna L Berry

DOI: https://doi.org/10.2196/23021

IF: 7.076

2020-10-07

Journal of Medical Internet Research

Abstract:Background Recruitment of health research participants through social media is becoming more common. In the United States, 80% of adults use at least one social media platform. Social media platforms may allow researchers to reach potential participants efficiently. However, online research methods may be associated with unique threats to sample validity and data integrity. Limited research has described issues of data quality and authenticity associated with the recruitment of health research participants through social media, and sources of low-quality and fraudulent data in this context are poorly understood. Objective The goal of the research was to describe and explain threats to sample validity and data integrity following recruitment of health research participants through social media and summarize recommended strategies to mitigate these threats. Our experience designing and implementing a research study using social media recruitment and online data collection serves as a case study. Methods Using published strategies to preserve data integrity, we recruited participants to complete an online survey through the social media platforms Twitter and Facebook. Participants were to receive $15 upon survey completion. Prior to manually issuing remuneration, we reviewed completed surveys for indicators of fraudulent or low-quality data. Indicators attributable to respondent error were labeled suspicious, while those suggesting misrepresentation were labeled fraudulent. We planned to remove cases with 1 fraudulent indicator or at least 3 suspicious indicators. Results Within 7 hours of survey activation, we received 271 completed surveys. We classified 94.5% (256/271) of cases as fraudulent and 5.5% (15/271) as suspicious. In total, 86.7% (235/271) provided inconsistent responses to verifiable items and 16.2% (44/271) exhibited evidence of bot automation. Of the fraudulent cases, 53.9% (138/256) provided a duplicate or unusual response to one or more open-ended items and 52.0% (133/256) exhibited evidence of inattention. Conclusions Research findings from several disciplines suggest studies in which research participants are recruited through social media are susceptible to data quality issues. Opportunistic individuals who use virtual private servers to fraudulently complete research surveys for profit may contribute to low-quality data. Strategies to preserve data integrity following research participant recruitment through social media are limited. Development and testing of novel strategies to prevent and detect fraud is a research priority.

health care sciences & services,medical informatics
Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire (Preprint)

Rachel Pozzar,Marilyn J Hammer,Meghan Underhill-Blazey,Alexi A Wright,James A Tulsky,Fangxin Hong,Daniel A Gundersen,Donna L Berry

DOI: https://doi.org/10.2196/preprints.23021

2020-07-30

Abstract:BACKGROUND Recruitment of health research participants through social media is becoming more common. In the United States, 80% of adults use at least one social media platform. Social media platforms may allow researchers to reach potential participants efficiently. However, online research methods may be associated with unique threats to sample validity and data integrity. Limited research has described issues of data quality and authenticity associated with the recruitment of health research participants through social media, and sources of low-quality and fraudulent data in this context are poorly understood. OBJECTIVE The goal of the research was to describe and explain threats to sample validity and data integrity following recruitment of health research participants through social media and summarize recommended strategies to mitigate these threats. Our experience designing and implementing a research study using social media recruitment and online data collection serves as a case study. METHODS Using published strategies to preserve data integrity, we recruited participants to complete an online survey through the social media platforms Twitter and Facebook. Participants were to receive $15 upon survey completion. Prior to manually issuing remuneration, we reviewed completed surveys for indicators of fraudulent or low-quality data. Indicators attributable to respondent error were labeled suspicious, while those suggesting misrepresentation were labeled fraudulent. We planned to remove cases with 1 fraudulent indicator or at least 3 suspicious indicators. RESULTS Within 7 hours of survey activation, we received 271 completed surveys. We classified 94.5% (256/271) of cases as fraudulent and 5.5% (15/271) as suspicious. In total, 86.7% (235/271) provided inconsistent responses to verifiable items and 16.2% (44/271) exhibited evidence of bot automation. Of the fraudulent cases, 53.9% (138/256) provided a duplicate or unusual response to one or more open-ended items and 52.0% (133/256) exhibited evidence of inattention. CONCLUSIONS Research findings from several disciplines suggest studies in which research participants are recruited through social media are susceptible to data quality issues. Opportunistic individuals who use virtual private servers to fraudulently complete research surveys for profit may contribute to low-quality data. Strategies to preserve data integrity following research participant recruitment through social media are limited. Development and testing of novel strategies to prevent and detect fraud is a research priority.
Comparing Health Survey Data Cost and Quality Between Amazon's Mechanical Turk and Ipsos' KnowledgePanel: Observational Study

Patricia M Herman,Mary E Slaughter,Nabeel Qureshi,Tarek Azzam,David Cella,Ian D Coulter,Graham DiGuiseppi,Maria Orlando Edelen,Arie Kapteyn,Anthony Rodriguez,Max Rubinstein,Ron D Hays

DOI: https://doi.org/10.2196/63032

IF: 7.076

2024-11-30

Journal of Medical Internet Research

Abstract:Background: Researchers have many options for web-based survey data collection, ranging from access to curated probability-based panels, where individuals are selectively invited to join based on their membership in a representative population, to convenience panels, which are open for anyone to join. The mix of respondents available also varies greatly regarding representation of a population of interest and in motivation to provide thoughtful and accurate responses. Despite the additional dataset-building labor required of the researcher, convenience panels are much less expensive than probability-based panels. However, it is important to understand what may be given up regarding data quality for those cost savings. Objective: This study examined the relative costs and data quality of fielding equivalent surveys on Amazon's Mechanical Turk (MTurk), a convenience panel, and KnowledgePanel, a nationally representative probability-based panel. Methods: We administered the same survey measures to MTurk (in 2021) and KnowledgePanel (in 2022) members. We applied several recommended quality assurance steps to enhance the data quality achieved using MTurk. Ipsos, the owner of KnowledgePanel, followed their usual (industry standard) protocols. The survey was designed to support psychometric analyses and included >60 items from the Patient-Reported Outcomes Measurement Information System (PROMIS), demographics, and a list of health conditions. We used 2 fake conditions ("syndomitis" and "chekalism") to identify those more likely to be honest respondents. We examined the quality of each platform's data using several recommended metrics (eg, consistency, reliability, representativeness, missing data, and correlations) including and excluding those respondents who had endorsed a fake condition and examined the impact of weighting on representativeness. Results: We found that prescreening in the MTurk sample (removing those who endorsed a fake health condition) improved data quality but KnowledgePanel data quality generally remained superior. While MTurk's unweighted point estimates for demographics exhibited the usual mismatch with national averages (younger, better educated, and lower income), weighted MTurk data matched national estimates. KnowledgePanel's point estimates better matched national benchmarks even before poststratification weighting. Correlations between PROMIS measures and age and income were similar in MTurk and KnowledgePanel; the mean absolute value of the difference between each platform's 137 correlations was 0.06, and 92% were <0.15. However, correlations between PROMIS measures and educational level were dramatically different; the mean absolute value of the difference across these 17 correlation pairs was 0.15, the largest difference was 0.29, and the direction of more than half of these relationships in the MTurk sample was the opposite from that expected from theory. Therefore, caution is needed if using MTurk for studies where educational level is a key variable. Conclusions: The data quality of our MTurk sample was often inferior to that of the KnowledgePanel sample but possibly not so much as to negate the benefits of its cost savings for some uses.

health care sciences & services,medical informatics
How Well Do My Results Generalize Now? The External Validity of Online Privacy and Security Surveys

Jenny Tang,Eleanor Birrell,Ada Lerner

DOI: https://doi.org/10.48550/arXiv.2202.14036

IF: 6.4588

2022-02-28

Human-Computer Interaction

Abstract:Privacy and security researchers often rely on data collected through online crowdsourcing platforms such as Amazon Mechanical Turk (MTurk) and Prolific. Prior work -- which used data collected in the United States between 2013 and 2017 -- found that MTurk responses regarding security and privacy were generally representative for people under 50 or with some college education. However, the landscape of online crowdsourcing has changed significantly over the last five years, with the rise of Prolific as a major platform and the increasing presence of bots. This work attempts to replicate the prior results about the external validity of online privacy and security surveys. We conduct an online survey on MTurk (n=800), a gender-balanced survey on Prolific (n=800), and a representative survey on Prolific (n=800) and compare the responses to a probabilistic survey conducted by the Pew Research Center (n=4272). We find that MTurk response quality has degraded over the last five years, and our results do not replicate the earlier finding about the generalizability of MTurk responses. By contrast, we find that data collected through Prolific is generally representative for questions about user perceptions and experiences, but not for questions about security and privacy knowledge. We also evaluate the impact of Prolific settings, attention check questions, and statistical methods on the external validity of online surveys, and we develop recommendations about best practices for conducting online privacy and security surveys.
Quality control questions on Amazon’s Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7

Jon Agley,Yunyu Xiao,Rachael Nolan,Lilian Golzarri-Arroyo

DOI: https://doi.org/10.3758/s13428-021-01665-8

IF: 5.953

2021-08-06

Behavior Research Methods

Abstract:Abstract Crowdsourced psychological and other biobehavioral research using platforms like Amazon’s Mechanical Turk (MTurk) is increasingly common – but has proliferated more rapidly than studies to establish data quality best practices. Thus, this study investigated whether outcome scores for three common screening tools would be significantly different among MTurk workers who were subject to different sets of quality control checks. We conducted a single-stage, randomized controlled trial with equal allocation to each of four study arms: Arm 1 (Control Arm), Arm 2 (Bot/VPN Check), Arm 3 (Truthfulness/Attention Check), and Arm 4 (Stringent Arm – All Checks). Data collection was completed in Qualtrics, to which participants were referred from MTurk. Subjects ( n = 1100) were recruited on November 20–21, 2020. Eligible workers were required to claim U.S. residency, have a successful task completion rate > 95%, have completed a minimum of 100 tasks, and have completed a maximum of 10,000 tasks. Participants completed the US-Alcohol Use Disorders Identification Test (USAUDIT), the Patient Health Questionnaire (PHQ-9), and a screener for Generalized Anxiety Disorder (GAD-7). We found that differing quality control approaches significantly, meaningfully, and directionally affected outcome scores on each of the screening tools. Most notably, workers in Arm 1 (Control) reported higher scores than those in Arms 3 and 4 for all tools, and a higher score than workers in Arm 2 for the PHQ-9. These data suggest that the use, or lack thereof, of quality control questions in crowdsourced research may substantively affect findings, as might the types of quality control items.

psychology, experimental, mathematical
Did a bot eat your homework? An assessment of the potential impact of bad actors in online administration of preference surveys

Juan Marcos Gonzalez,Kiran Grover,Thomas W Leblanc,Bryce B Reeve

DOI: https://doi.org/10.1371/journal.pone.0287766

IF: 3.7

2023-10-05

PLoS ONE

Abstract:Background: Online administration of surveys has a number of advantages but can also lead to increased exposure to bad actors (human and non-human bots) who can try to influence the study results or to benefit financially from the survey. We analyze data collected through an online discrete-choice experiment (DCE) survey to evaluate the likelihood that bad actors can affect the quality of the data collected. Methods: We developed and fielded a survey instrument that included two sets of DCE questions asking respondents to select their preferred treatments for multiple myeloma therapies. The survey also included questions to assess respondents' attention while completing the survey and their understanding of the DCE questions. We used a latent-class model to identify a class associated with perverse preferences or high model variance, and the degree to which the quality checks included in the survey were correlated with class membership. Class-membership probabilities for the problematic class were used as weights in a random-parameters logit to recover population-level estimates that minimizes exposure to potential bad actors. Results: Results show a significant proportion of respondents provided answers with a high degree of variability consistent with responses from bad actors. We also found that a wide-ranging selection of conditions in the survey screener is more consistent with choice patterns expected from bad actors looking to qualify for the study. The relationship between the number of incorrect answers to comprehension questions and problematic choice patterns peaked around 5 out of 10 questions. Conclusions: Our results highlight the need for a robust discussion around the appropriate way to handle bad actors in online preference surveys. While exclusion of survey respondents must be avoided under most circumstances, the impact of "bots" on preference estimates can be significant.
Running experiments on Amazon Mechanical Turk

Gabriele Paolacci,Jesse Chandler,Panagiotis G. Ipeirotis

DOI: https://doi.org/10.1017/s1930297500002205

2010-08-01

Judgment and decision making

Abstract:Abstract Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.

psychology, multidisciplinary
Improving data quality in online parenting research

Makena L. Kaylor‐Tapscott,Maddison N. Tolliver‐Lynn,Maureen A. Sullivan

DOI: https://doi.org/10.1002/icd.2525

2024-06-12

Infant and Child Development

Abstract:Online surveys provide many benefits for researchers but are vulnerable to fraudulent responders that threaten data quality. We aimed to evaluate the effectiveness of current detection strategies after noting patterns in our data. We systematically reviewed responses (N = 180) from parenting studies utilizing two online platforms. Respondents were categorized as accepted (n = 45) or fraudulent (n = 135), and responses on standardized measures were examined to determine if fraudulent scores differed. Accepted participants were predominately female (98.4%) and White (67.9%). Average caregiver age was 40.61 (SD = 7.37) on Qualtrics and 59.88 (SD = 6.01) on REDCap. The majority of fraudulent responders reported being female (62.1%) and White (80.6%), with average caregiver ages of 37.83 (SD = 7.35) for Qualtrics and 56.16 (SD = 8.86) for REDCap. Fraudulent responders had significantly higher median scores on measures of caregiver distress (rDASS = 0.21) and child behavioural problems (rCBCL = 0.33). Previously recommended strategies were insufficient at preventing and identifying fraudulent responders. Recommendations for improving these strategies are discussed. Highlights The current paper includes experiences using multiple strategies to detect fraudulent responders in two online parenting studies. Improvements to previously recommended strategies and an example review form are included to aid researchers in detecting fraudulent responders.

psychology, developmental
Amazon's Mechanical Turk

Michael Buhrmester,Tracy Kwang,Samuel D. Gosling

DOI: https://doi.org/10.1177/1745691610393980

2011-01-01

Perspectives on Psychological Science

Abstract:Amazon’s Mechanical Turk (MTurk) is a relatively new website that contains the major elements required to conduct research: an integrated participant compensation system; a large participant pool; and a streamlined process of study design, participant recruitment, and data collection. In this article, we describe and evaluate the potential contributions of MTurk to psychology and other social sciences. Findings indicate that (a) MTurk participants are slightly more demographically diverse than are standard Internet samples and are significantly more diverse than typical American college samples; (b) participation is affected by compensation rate and task length, but participants can still be recruited rapidly and inexpensively; (c) realistic compensation rates do not affect data quality; and (d) the data obtained are at least as reliable as those obtained via traditional methods. Overall, MTurk can be used to obtain high-quality data inexpensively and rapidly.

psychology, multidisciplinary
The Hidden Cost of Using Amazon Mechanical Turk for Research

Antonios Saravanos,Stavros Zervoudakis,Dongnanzi Zheng,Neil Stott,Bohdan Hawryluk,Donatella Delfino

DOI: https://doi.org/10.1007/978-3-030-90238-4_12

2021-06-10

Abstract:In this study, we investigate the attentiveness exhibited by participants sourced through Amazon Mechanical Turk (MTurk), thereby discovering a significant level of inattentiveness amongst the platform's top crowd workers (those classified as 'Master', with an 'Approval Rate' of 98% or more, and a 'Number of HITS approved' value of 1,000 or more). A total of 564 individuals from the United States participated in our experiment. They were asked to read a vignette outlining one of four hypothetical technology products and then complete a related survey. Three forms of attention check (logic, honesty, and time) were used to assess attentiveness. Through this experiment we determined that a total of 126 (22.3%) participants failed at least one of the three forms of attention check, with most (94) failing the honesty check - followed by the logic check (31), and the time check (27). Thus, we established that significant levels of inattentiveness exist even among the most elite MTurk workers. The study concludes by reaffirming the need for multiple forms of carefully crafted attention checks, irrespective of whether participant quality is presumed to be high according to MTurk criteria such as 'Master', 'Approval Rate', and 'Number of HITS approved'. Furthermore, we propose that researchers adjust their proposals to account for the effort and costs required to address participant inattentiveness.

Human-Computer Interaction,Computers and Society
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup

Lu Lu,Nathan Neale,Nathaniel D. Line,Mark Bonn

DOI: https://doi.org/10.1177/19389655211025475

2021-07-01

Cornell Hospitality Quarterly

Abstract:As the use of Amazon’s Mechanical Turk (MTurk) has increased among social science researchers, so, too, has research into the merits and drawbacks of the platform. However, while many endeavors have sought to address issues such as generalizability, the attentiveness of workers, and the quality of the associated data, there has been relatively less effort concentrated on integrating the various strategies that can be used to generate high-quality data using MTurk samples. Accordingly, the purpose of this research is twofold. First, existing studies are integrated into a set of strategies/best practices that can be used to maximize MTurk data quality. Second, focusing on task setup, selected platform-level strategies that have received relatively less attention in previous research are empirically tested to further enhance the contribution of the proposed best practices for MTurk usage.

management,hospitality, leisure, sport & tourism,sociology
Participant Misrepresentation in Online Focus Groups: Red Flags and Proactive Measures

Lesley Andrew,Emily Gizzarelli,Mohamed Estai,Ruth Wallace

DOI: https://doi.org/10.1002/eahr.500198

Abstract:Covid-19 public health measures prompted a significant increase in online research. This approach has several benefits over face-to-face data-collection methods, including lower cost and wider geographical reach of participants. Yet when the online data-collection instrument is a survey, there are also well-documented drawbacks of participant misrepresentation and related data-authenticity issues. However, the scholarly literature has not looked at participant misrepresentation in online focus-group empirical research. This case study communicates a concerning situation that arose during our research project: dishonest participant behavior threatened the integrity and validity of our data collected through online focus-group sessions as well as e-surveys. We describe the study context, initial red flags alerting us to the issue, subsequent investigations, and implications for research ethics, funding, and data quality. We conclude with a discussion of potential steps to safeguard future online focus-group research against similar issues.
Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations

Janelle H. Cheung,Deanna K. Burns,Robert R. Sinclair,Michael Sliter

DOI: https://doi.org/10.1007/s10869-016-9458-5

IF: 6.604

2016-06-29

Journal of Business and Psychology

Abstract:PurposeAmazon Mechanical Turk is an increasingly popular data source in the organizational psychology research community. This paper presents an evaluation of MTurk and provides a set of practical recommendations for researchers using MTurk.Design/Methodology/ApproachWe present an evaluation of methodological concerns related to the use of MTurk and potential threats to validity inferences. Based on our evaluation, we also provide a set of recommendations to strengthen validity inferences using MTurk samples.FindingsAlthough MTurk samples can overcome some important validity concerns, there are other limitations researchers must consider in light of their research objectives. Researchers should carefully evaluate the appropriateness and quality of MTurk samples based on the different issues we discuss in our evaluation.ImplicationsThere is not a one-size-fits-all answer to whether MTurk is appropriate for a research study. The answer depends on the research questions and the data collection and analytic procedures adopted. The quality of the data is not defined by the data source per se, but rather the decisions researchers make during the stages of study design, data collection, and data analysis.Originality/ValueThe current paper extends the literature by evaluating MTurk in a more comprehensive manner than in prior reviews. Past review papers focused primarily on internal and external validity, with less attention paid to statistical conclusion and construct validity—which are equally important in making accurate inferences about research findings. This paper also provides a set of practical recommendations in addressing validity concerns when using MTurk.

psychology, applied,business
Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages

Aaron J Moss,Cheskie Rosenzweig,Jonathan Robinson,Shalom N Jaffe,Leib Litman

DOI: https://doi.org/10.3758/s13428-022-02005-0

Abstract:To understand human behavior, social scientists need people and data. In the last decade, Amazon's Mechanical Turk (MTurk) emerged as a flexible, affordable, and reliable source of human participants and was widely adopted by academics. Yet despite MTurk's utility, some have questioned whether researchers should continue using the platform on ethical grounds. The brunt of their concern is that people on MTurk are financially insecure, subject to abuse, and earn inhumane wages. We investigated these issues with two representative probability surveys of the U.S. MTurk population (N = 4094). The surveys revealed: (1) the financial situation of people on MTurk mirrors the general population, (2) most participants do not find MTurk stressful or requesters abusive, and (3) MTurk offers flexibility and benefits that most people value above other options for work. People reported it is possible to earn more than $10 per hour and said they would not trade the flexibility of MTurk for less than $25 per hour. Altogether, our data are important for assessing whether MTurk is an ethical place for research.
Lessons learned from the MOMENT study on how to recruit and retain a target population online, across borders, and with automated remote data collection

Andrew Belfiglio,Shayleigh Dickson Page,Sara Pettersson,Michelle van Rijn,Ercole Vellone,Heleen Westland,Kenneth E Freedland,Christopher Lee,Anna Strömberg,Douglas Wiebe,Subhash Aryal,Barbara Riegel,Tiny Jaarsma

DOI: https://doi.org/10.1371/journal.pone.0307440

IF: 3.7

2024-09-16

PLoS ONE

Abstract:Increasingly, studies use social media to recruit, enroll, and collect data from participants. This introduces a threat to data integrity: efforts to produce fraudulent data to receive participant compensation, e.g., gift cards. MOMENT is an online symptom-monitoring and self-care study that implemented safeguards to protect data integrity. Facebook, Twitter, and patient organizations were used to recruit participants with chronic health conditions in four countries (USA, Italy, The Netherlands, Sweden). Links to the REDCap baseline survey were posted to social media accounts. The initial study launch, where participants completed the baseline survey and were automatically re-directed to the LifeData ecological momentary assessment app, was overwhelmed with fraudulent responses. In response, safeguards (e.g., reCAPTCHA, attention checks) were implemented and baseline data was manually inspected prior to LifeData enrollment. The initial launch resulted in 411 responses in 48 hours, 265 of which (64.5%) successfully registered for the LifeData app and were considered enrolled. Ninety-nine percent of these were determined to be fraudulent. Following implementation of safeguards, the re-launch yielded 147 completed baselines in 3.5 months. Eighteen cases (12.2%) were found fraudulent and not invited to enroll. Most fraudulent cases in the re-launch (15 of 18) were identified by a single attention check question. In total, 96.1% of fraudulent responses were to the USA-based survey. Data integrity safeguards are necessary for research studies that recruit online and should be reported in manuscripts. Three safeguard strategies were effective in preventing and removing most of the fraudulent data in the MOMENT study. Additional strategies were also used and may be necessary in other contexts.
Methodological considerations for behavioral studies relying on response time outcomes through online crowdsourcing platforms

Patrick A. McConnell,Christian Finetto,Kirstin-Friederike Heise

DOI: https://doi.org/10.1038/s41598-024-58300-7

IF: 4.6

2024-04-03

Scientific Reports

Abstract:This perspective paper explores challenges associated with online crowdsourced data collection, particularly focusing on longitudinal tasks with time-sensitive outcomes like response latencies. Based on our research, we identify two significant sources of bias: technical shortcomings such as low, variable frame rates, and human factors, contributing to high attrition rates. We explored potential solutions to these problems, such as enforcing hardware acceleration and defining study-specific frame rate thresholds, as well as pre-screening participants and monitoring hardware performance and task engagement over each experimental session. With this discussion, we intend to provide recommendations on how to improve the quality and reliability of data collected via online crowdsourced platforms and emphasize the need for researchers to be cognizant of potential pitfalls in online research.

multidisciplinary sciences

Nonsensical results plague online social science tool

An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results

The Burden for High-Quality Online Data Collection Lies With Researchers, Not Recruitment Platforms

Data quality of platforms and panels for online behavioral research

Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA

Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire

Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire (Preprint)

Comparing Health Survey Data Cost and Quality Between Amazon's Mechanical Turk and Ipsos' KnowledgePanel: Observational Study

How Well Do My Results Generalize Now? The External Validity of Online Privacy and Security Surveys

Quality control questions on Amazon’s Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7

Did a bot eat your homework? An assessment of the potential impact of bad actors in online administration of preference surveys

Running experiments on Amazon Mechanical Turk

Improving data quality in online parenting research

Amazon's Mechanical Turk

The Hidden Cost of Using Amazon Mechanical Turk for Research

Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup

Participant Misrepresentation in Online Focus Groups: Red Flags and Proactive Measures

Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations

Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages

Lessons learned from the MOMENT study on how to recruit and retain a target population online, across borders, and with automated remote data collection

Methodological considerations for behavioral studies relying on response time outcomes through online crowdsourcing platforms