Abstract:The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an "algorithm-in-the-loop'' process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decisions. Society lacks both clear normative principles regarding how people should collaborate with algorithms as well as robust empirical evidence about how people do collaborate with algorithms. Given research suggesting that people struggle to interpret machine learning models and to incorporate them into their decisions---sometimes leading these models to produce unexpected outcomes---it is essential to consider how different ways of presenting models and structuring human-algorithm interactions affect the quality and type of decisions made. This paper contributes to such research in two ways. First, we posited three principles as essential to ethical and responsible algorithm-in-the-loop decision making. Second, through a controlled experimental study on Amazon Mechanical Turk, we evaluated whether people satisfy these principles when making predictions with the aid of a risk assessment. We studied human predictions in two contexts (pretrial release and financial lending) and under several conditions for risk assessment presentation and structure. Although these conditions did influence participant behaviors and in some cases improved performance, only one desideratum was consistently satisfied. Under all conditions, our study participants 1) were unable to effectively evaluate the accuracy of their own or the risk assessment's predictions, 2) did not calibrate their reliance on the risk assessment based on the risk assessment's performance, and 3) exhibited bias in their interactions with the risk assessment. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact.

Facets of algorithmic literacy: Information, experience, and individual factors predict attitudes toward algorithmic systems

Experiencing Algorithms: How Young People Understand, Feel About, and Engage With Algorithmic News Selection on Social Media

When the Automated Fire Backfires: the Adoption of Algorithm-based HR Decision-making Could Induce Consumer's Unfavorable Ethicality Inferences of the Company

Factors Influencing Perceived Fairness in Algorithmic Decision-Making: Algorithm Outcomes, Development Procedures, and Individual Differences

Are algorithms always fair? The study on public preferences toward algorithmic decision‐making: A case study from the perspectives of decision scenarios and social roles

Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature

Designing for Critical Algorithmic Literacies

User Perceptions of Algorithmic Decisions in the Personalized AI System:Perceptual Evaluation of Fairness, Accountability, Transparency, and Explainability

Why do people resist algorithms? From the perspective of short video usage motivations

Perceived algorithmic fairness: An empirical study of transparency and anthropomorphism in algorithmic recruiting

Development and Validation of an Algorithm Literacy Scale for Internet Users

How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface

Communicating and combating algorithmic bias: effects of data diversity, labeler diversity, performance bias, and user feedback on AI trust

Unpacking the effects of personality traits on algorithmic awareness: The mediating role of previous knowledge and moderating role of internet use

Dimensions of Diversity in Human Perceptions of Algorithmic Fairness

Beyond user experience: What constitutes algorithmic experiences?

An Algorithmic Equity Toolkit for Technology Audits by Community Advocates and Activists

The Principles and Limits of Algorithm-in-the-Loop Decision Making

Automating Automaticity: How the Context of Human Choice Affects the Extent of Algorithmic Bias

"Public(s)-in-the-Loop": Facilitating Deliberation of Algorithmic Decisions in Contentious Public Policy Domains