AI Oversight and Human Mistakes: Evidence from Centre Court

David Almog,Romain Gauriot,Lionel Page,Daniel Martin

2024-02-19

Abstract:Powered by the increasing predictive capabilities of machine learning algorithms, artificial intelligence (AI) systems have begun to be used to overrule human mistakes in many settings. We provide the first field evidence this AI oversight carries psychological costs that can impact human decision-making. We investigate one of the highest visibility settings in which AI oversight has occurred: the Hawk-Eye review of umpires in top tennis tournaments. We find that umpires lowered their overall mistake rate after the introduction of Hawk-Eye review, in line with rational inattention given psychological costs of being overruled by AI. We also find that umpires increased the rate at which they called balls in, which produced a shift from making Type II errors (calling a ball out when in) to Type I errors (calling a ball in when out). We structurally estimate the psychological costs of being overruled by AI using a model of rational inattentive umpires, and our results suggest that because of these costs, umpires cared twice as much about Type II errors under AI oversight.

Machine Learning,Computers and Society,General Economics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how the behavior of human decision - makers changes after the introduction of artificial intelligence (AI) supervision, especially in high - visibility environments, such as referee decisions in top - level tennis matches. Specifically, the research focuses on whether an AI system (such as Hawk - Eye technology in tennis matches) used to correct human errors will cause psychological costs to human decision - makers and affect their decision - making behavior.

### Core issues of the paper
1. **Psychological costs of AI supervision**: The research explores whether AI supervision has caused psychological costs to human decision - makers, such as feelings of shame and embarrassment after being corrected by AI.
2. **Changes in decision - making behavior**: The research analyzes whether the decision - making behavior of human decision - makers has changed under AI supervision, especially whether the types of errors made by referees when judging whether the ball is out of bounds have changed.
3. **Shift in error types**: The research finds that under AI supervision, referees are more inclined to reduce Type II errors (i.e., judging a ball that is actually in - bounds as out - of - bounds), while increasing Type I errors (i.e., judging a ball that is actually out - of - bounds as in - bounds).

### Research background
- **Application of AI supervision**: With the improvement of the prediction ability of machine - learning algorithms, AI systems have begun to be used to correct human errors in multiple fields, especially in cases where a final decision needs to be made quickly or when humans ignore AI suggestions.
- **Hawk - Eye in tennis matches**: Hawk - Eye technology is used to challenge referee decisions in tennis matches. When a player's challenge is successful, the referee's decision will be overturned by the AI system.

### Main findings
- **Overall error rate decline**: The research finds that after the introduction of Hawk - Eye technology, the overall error rate of referees has decreased by 8% (1.1 percentage points), which is in line with the rational inattention theory, that is, the psychological cost of being corrected by AI prompts referees to be more cautious.
- **Shift in error types**: For balls closest to the sideline (with an error within 20 millimeters), the error rate of referees has actually increased by 22.9% (7.3 percentage points). This is because referees, in order to reduce Type II errors, more often judge the ball as in - bounds, resulting in an increase in Type I errors.
- **Estimation of psychological costs**: By constructing a rational inattention model, the research estimates the psychological cost of being corrected by AI, and the results show that referees under AI supervision pay twice as much attention to Type II errors as before.

### Conclusion
Although AI supervision can theoretically improve decision - making quality, in practical applications, it may have unexpected negative impacts due to psychological costs. The research emphasizes that when introducing AI supervision, it is necessary to consider its impact on the psychology and behavior of human decision - makers to avoid potential negative consequences.

### Related literature
- **AI and human decision - making**: Recent research shows that AI has surpassed human experts in many prediction tasks, but in some cases, humans are still able to integrate private information and understand the context, which are advantages that are difficult for AI to replace.
- **Social image and peer effects**: The research also contributes to the literature on social image and peer effects, showing that the cost of being corrected in public may exceed the potential incentives brought by technology.

### Data sources
- **Hawk - Eye basic data set**: It contains detailed data of international professional matches between 2005 and 2009, including all bounce positions of each point, the serving player, the current score, and the scorer.
- **Challenge data set**: It records the challenge results in matches in the years after the introduction of Hawk - Eye technology.
- **Video replay**: By manually reviewing match videos, the accuracy of the challenge data set is verified, and the best data merging method is determined.

Through these data, the research comprehensively evaluates the impact of AI supervision on the decision - making behavior of tennis referees.