Abstract:The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an "algorithm-in-the-loop'' process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decisions. Society lacks both clear normative principles regarding how people should collaborate with algorithms as well as robust empirical evidence about how people do collaborate with algorithms. Given research suggesting that people struggle to interpret machine learning models and to incorporate them into their decisions---sometimes leading these models to produce unexpected outcomes---it is essential to consider how different ways of presenting models and structuring human-algorithm interactions affect the quality and type of decisions made. This paper contributes to such research in two ways. First, we posited three principles as essential to ethical and responsible algorithm-in-the-loop decision making. Second, through a controlled experimental study on Amazon Mechanical Turk, we evaluated whether people satisfy these principles when making predictions with the aid of a risk assessment. We studied human predictions in two contexts (pretrial release and financial lending) and under several conditions for risk assessment presentation and structure. Although these conditions did influence participant behaviors and in some cases improved performance, only one desideratum was consistently satisfied. Under all conditions, our study participants 1) were unable to effectively evaluate the accuracy of their own or the risk assessment's predictions, 2) did not calibrate their reliance on the risk assessment based on the risk assessment's performance, and 3) exhibited bias in their interactions with the risk assessment. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact.

Modelling dataset bias in machine-learned theories of economic decision-making

Predicting human decisions with behavioral theories and machine learning

Using recurrent neural network to estimate irreducible stochasticity in human choice behavior

Behavior-Based Machine-Learning: A Hybrid Approach for Predicting Human Decision Making

The Principles and Limits of Algorithm-in-the-Loop Decision Making

The Risk of Machine Learning

Uncovering the Source of Machine Bias

Dynamic noise estimation: A generalized method for modeling noise fluctuations in decision-making

Scaling up Psychology via Scientific Regret Minimization: A Case Study in Moral Decisions

Identifying and examining machine learning biases on Adult dataset

Neural Mechanism Underlying Risk Attitude and Probability Distortion: One Two-Stage Model of Valuation and Choice

Using large-scale experiments and machine learning to discover theories of human decision-making

Decisions Under Uncertainty as Bayesian Inference on Choice Options

Using Machine Learning to Guide Cognitive Modeling: A Case Study in Moral Reasoning

Capturing the Complexity of Human Strategic Decision-Making with Machine Learning

My Model is Unfair, Do People Even Care? Visual Design Affects Trust and Perceived Bias in Machine Learning

Limited or Biased: Modeling Sub-Rational Human Investors in Financial Markets

Contradictory deviations from maximization: Environment-specific biases, or reflections of basic properties of human learning?

Facing the Challenges of Developing Fair Risk Scoring Models

Apparent sunk cost effect in rational agents