Abstract:Background: Online administration of surveys has a number of advantages but can also lead to increased exposure to bad actors (human and non-human bots) who can try to influence the study results or to benefit financially from the survey. We analyze data collected through an online discrete-choice experiment (DCE) survey to evaluate the likelihood that bad actors can affect the quality of the data collected. Methods: We developed and fielded a survey instrument that included two sets of DCE questions asking respondents to select their preferred treatments for multiple myeloma therapies. The survey also included questions to assess respondents' attention while completing the survey and their understanding of the DCE questions. We used a latent-class model to identify a class associated with perverse preferences or high model variance, and the degree to which the quality checks included in the survey were correlated with class membership. Class-membership probabilities for the problematic class were used as weights in a random-parameters logit to recover population-level estimates that minimizes exposure to potential bad actors. Results: Results show a significant proportion of respondents provided answers with a high degree of variability consistent with responses from bad actors. We also found that a wide-ranging selection of conditions in the survey screener is more consistent with choice patterns expected from bad actors looking to qualify for the study. The relationship between the number of incorrect answers to comprehension questions and problematic choice patterns peaked around 5 out of 10 questions. Conclusions: Our results highlight the need for a robust discussion around the appropriate way to handle bad actors in online preference surveys. While exclusion of survey respondents must be avoided under most circumstances, the impact of "bots" on preference estimates can be significant.

The Behavioral Foundations of Default Effects: Theory and Evidence from Medicare Part D

Defaults are not a panacea: distinguishing between default effects on choices and on outcomes

The impact of experience on the tendency to accept recommended defaults

Framing the Default Option Right

Inattention and Switching Costs as Sources of Inertia in Medicare Part D

Nudge for Good? Choice Defaults and Spillover Effects

The application of defaults to optimize parents' health-based choices for children

Defaults: a double-edged sword in governing common resources

Moral Constraints, Social Norm Enforcement, and Strategic Default in Weak and Strong Economic Conditions

Persistent Effects of Temporary Incentives: Evidence from a Nationwide Health Insurance Experiment

Disentangling Moral Hazard and Adverse Selection in Private Health Insurance

Anticipated Benefit Termination and Health Care Consumption Responses: Evidence from a Quasi-Experiment

Subsidy Design in Privately Provided Social Insurance: Lessons from Medicare Part D

VOLUNTARY REGULATION: EVIDENCE FROM MEDICARE PAYMENT REFORM

Bridging the gap: Experimental evidence on information provision and health insurance choices

Did a bot eat your homework? An assessment of the potential impact of bad actors in online administration of preference surveys

Choice overload paradox and public policy design. The case of Swedish pension system

“Buy Now, Pay Later” (BNPL): Optimizing Customer Targeting Decisions Using Payment Default and Product Return Option Values

Adverse selection and consumer inertia: empirical evidence from the Dutch health insurance market

Selection, Leverage, and Default in the Mortgage Market

Optimal Defaults in Online Grocery Shopping: A Secondary Analysis to Explore Impacts in Multiresident Households and Families