Abstract:The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an "algorithm-in-the-loop'' process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decisions. Society lacks both clear normative principles regarding how people should collaborate with algorithms as well as robust empirical evidence about how people do collaborate with algorithms. Given research suggesting that people struggle to interpret machine learning models and to incorporate them into their decisions---sometimes leading these models to produce unexpected outcomes---it is essential to consider how different ways of presenting models and structuring human-algorithm interactions affect the quality and type of decisions made. This paper contributes to such research in two ways. First, we posited three principles as essential to ethical and responsible algorithm-in-the-loop decision making. Second, through a controlled experimental study on Amazon Mechanical Turk, we evaluated whether people satisfy these principles when making predictions with the aid of a risk assessment. We studied human predictions in two contexts (pretrial release and financial lending) and under several conditions for risk assessment presentation and structure. Although these conditions did influence participant behaviors and in some cases improved performance, only one desideratum was consistently satisfied. Under all conditions, our study participants 1) were unable to effectively evaluate the accuracy of their own or the risk assessment's predictions, 2) did not calibrate their reliance on the risk assessment based on the risk assessment's performance, and 3) exhibited bias in their interactions with the risk assessment. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact.

Keeping Designers in the Loop: Communicating Inherent Algorithmic Trade-offs Across Multiple Objectives

When the Automated Fire Backfires: the Adoption of Algorithm-based HR Decision-making Could Induce Consumer's Unfavorable Ethicality Inferences of the Company

Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders

The Principles and Limits of Algorithm-in-the-Loop Decision Making

Designing Equitable Algorithms

Designing Algorithmic Recommendations to Achieve Human-AI Complementarity

Are We Asking the Right Questions?: Designing for Community Stakeholders' Interactions with AI in Policing

'Transparency is Meant for Control' and Vice Versa: Learning from Co-designing and Evaluating Algorithmic News Recommenders

Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction

Who is the Expert? Reconciling Algorithm Aversion and Algorithm Appreciation in AI-Supported Decision Making

Transparency as design publicity: explaining and justifying inscrutable algorithms

Factors Influencing Perceived Fairness in Algorithmic Decision-Making: Algorithm Outcomes, Development Procedures, and Individual Differences

Are algorithms always fair? The study on public preferences toward algorithmic decision‐making: A case study from the perspectives of decision scenarios and social roles

Persuasion, Delegation, and Private Information in Algorithm-Assisted Decisions

Algorithm Design: A Fairness-Accuracy Frontier

Integrating Behavioral, Economic, and Technical Insights to Understand and Address Algorithmic Bias: A Human-Centric Perspective

WeBuildAI

Examining the Impact of Algorithm Awareness on Wikidata's Recommender System Recoin

Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirement

The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop Failures

LUCID: Exposing Algorithmic Bias through Inverse Design