Abstract:The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an "algorithm-in-the-loop'' process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decisions. Society lacks both clear normative principles regarding how people should collaborate with algorithms as well as robust empirical evidence about how people do collaborate with algorithms. Given research suggesting that people struggle to interpret machine learning models and to incorporate them into their decisions---sometimes leading these models to produce unexpected outcomes---it is essential to consider how different ways of presenting models and structuring human-algorithm interactions affect the quality and type of decisions made. This paper contributes to such research in two ways. First, we posited three principles as essential to ethical and responsible algorithm-in-the-loop decision making. Second, through a controlled experimental study on Amazon Mechanical Turk, we evaluated whether people satisfy these principles when making predictions with the aid of a risk assessment. We studied human predictions in two contexts (pretrial release and financial lending) and under several conditions for risk assessment presentation and structure. Although these conditions did influence participant behaviors and in some cases improved performance, only one desideratum was consistently satisfied. Under all conditions, our study participants 1) were unable to effectively evaluate the accuracy of their own or the risk assessment's predictions, 2) did not calibrate their reliance on the risk assessment based on the risk assessment's performance, and 3) exhibited bias in their interactions with the risk assessment. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact.

When the Oracle Misleads: Modeling the Consequences of Using Observable Rather than Potential Outcomes in Risk Assessment Instruments

Why Can't We Accurately Predict Others' Decisions? Prediction Discrepancy in Risky Decision-Making

The Impact of Algorithmic Risk Assessments on Human Predictions and its Analysis via Crowdsourcing Studies

Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contexts

Fairness in Risk Assessment Instruments: Post-Processing to Achieve Counterfactual Equalized Odds

The Progression of Disparities within the Criminal Justice System: Differential Enforcement and Risk Assessment Instruments

Counterfactual risk assessments, evaluation, and fairness

Counterfactual Prediction Under Outcome Measurement Error

Interventions over Predictions: Reframing the Ethical Debate for Actuarial Risk Assessment

Algorithmic Decision Making in the Presence of Unmeasured Confounding

Feedback Effects in Repeat-Use Criminal Risk Assessments

People Perceive Algorithmic Assessments as Less Fair and Trustworthy Than Identical Human Assessments

"If it didn't happen, why would I change my decision?": How Judges Respond to Counterfactual Explanations for the Public Safety Assessment

Risk Scores, Label Bias, and Everything but the Kitchen Sink

Risk scores, label bias, everything but the kitchen sink

Almost Politically Acceptable Criminal Justice Risk Assessment

Modeling and Interpreting Real-world Human Risk Decision Making with Inverse Reinforcement Learning

When accurate prediction models yield harmful self-fulfilling prophecies

The Principles and Limits of Algorithm-in-the-Loop Decision Making

Identifying Prediction Mistakes in Observational Data

Flipping the Script on Criminal Justice Risk Assessment: An actuarial model for assessing the risk the federal sentencing system poses to defendants