Abstract:The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an "algorithm-in-the-loop'' process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decisions. Society lacks both clear normative principles regarding how people should collaborate with algorithms as well as robust empirical evidence about how people do collaborate with algorithms. Given research suggesting that people struggle to interpret machine learning models and to incorporate them into their decisions---sometimes leading these models to produce unexpected outcomes---it is essential to consider how different ways of presenting models and structuring human-algorithm interactions affect the quality and type of decisions made. This paper contributes to such research in two ways. First, we posited three principles as essential to ethical and responsible algorithm-in-the-loop decision making. Second, through a controlled experimental study on Amazon Mechanical Turk, we evaluated whether people satisfy these principles when making predictions with the aid of a risk assessment. We studied human predictions in two contexts (pretrial release and financial lending) and under several conditions for risk assessment presentation and structure. Although these conditions did influence participant behaviors and in some cases improved performance, only one desideratum was consistently satisfied. Under all conditions, our study participants 1) were unable to effectively evaluate the accuracy of their own or the risk assessment's predictions, 2) did not calibrate their reliance on the risk assessment based on the risk assessment's performance, and 3) exhibited bias in their interactions with the risk assessment. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact.

Correcting User Decisions Based on Incorrect Machine Learning Decisions

Effects of machine learning errors on human decision-making: manipulations of model accuracy, error types, and error importance

Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks

On the Fairness of Machine-Assisted Human Decisions

HUMAN DECISIONS AND MACHINE PREDICTIONS

Human and Machine: The Impact of Machine Input on Decision-Making Under Cognitive Limitations

The Principles and Limits of Algorithm-in-the-Loop Decision Making

Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy

Learning When to Advise Human Decision Makers

The impact of AI errors in a human-in-the-loop process

To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems

Incorporating Experts' Judgment into Machine Learning Models

On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection

Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models

Human and Machine: The Impact of Machine Input on Decision Making Under Cognitive Limitations

The Prevalence of Errors in Machine Learning Experiments

Validity Concerns of Using Machine Learning in Management Research

Putting a human in the loop: Increasing uptake, but decreasing accuracy of automated decision-making

Between accurate prediction and poor decision making: the AI/ML gap

Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making

Human-Algorithm Interaction Biases in the Big Data Cycle: A Markov Chain Iterated Learning Framework