An Ecologically Valid Approach to Evaluating Online Gatekeepers

Sajad Shirali-Shahreza,Gerald Penn
DOI: https://doi.org/10.1080/10447318.2024.2398890
IF: 4.92
2024-09-13
International Journal of Human-Computer Interaction
Abstract:CAPTCHAs are commonly used as gatekeepers to protect online services from automated bots. Previous CAPTCHA evaluations have solely focussed on whether CAPTCHA challenges are unbreakable by bots but solvable by humans and have ignored a fundamental question: what effect do these gatekeepers have on their human users? Our study proposes a more realistic approach that complements existing CAPTCHA evaluations by measuring CAPTCHA's effect on the main task that the user wants to complete. Through our experimental setup, we show that failing to correctly answer CAPTCHAs can turn away a significant percentage of legitimate, human users. We also show that these failures have temporary knock-on effects on the quality of tasks that users later perform. Methodologically, our study also reveals limitations of current approaches to batch evaluations of CAPTCHAs that do not accurately capture the effects that ordering and variance in difficulty have on users' CAPTCHA-solving performance.
computer science, cybernetics,ergonomics
What problem does this paper attempt to address?