On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives

Sarah Sterz,Kevin Baum,Sebastian Biewer,Holger Hermanns,Anne Lauber-Rönsberg,Philip Meinel,Markus Langer

DOI: https://doi.org/10.1145/3630106.3659051

2024-05-07

Abstract:Human oversight is currently discussed as a potential safeguard to counter some of the negative aspects of high-risk AI applications. This prompts a critical examination of the role and conditions necessary for what is prominently termed effective or meaningful human oversight of these systems. This paper investigates effective human oversight by synthesizing insights from psychological, legal, philosophical, and technical domains. Based on the claim that the main objective of human oversight is risk mitigation, we propose a viable understanding of effectiveness in human oversight: for human oversight to be effective, the oversight person has to have (a) sufficient causal power with regard to the system and its effects, (b) suitable epistemic access to relevant aspects of the situation, (c) self-control, and (d) fitting intentions for their role. Furthermore, we argue that this is equivalent to saying that an oversight person is effective if and only if they are morally responsible and have fitting intentions. Against this backdrop, we suggest facilitators and inhibitors of effectiveness in human oversight when striving for practical applicability. We discuss factors in three domains, namely, the technical design of the system, individual factors of oversight persons, and the environmental circumstances in which they operate. Finally, this paper scrutinizes the upcoming AI Act of the European Union -- in particular Article 14 on Human Oversight -- as an exemplary regulatory framework in which we study the practicality of our understanding of effective human oversight. By analyzing the provisions and implications of the European AI Act proposal, we pinpoint how far that proposal aligns with our analyses regarding effective human oversight as well as how it might get enriched by our conceptual understanding of effectiveness in human oversight.

Computers and Society

What problem does this paper attempt to address?

The core problem that this paper attempts to solve is: **In high - risk AI applications, how to achieve effective human oversight?** Specifically, the authors explored under which conditions human oversight can be considered effective and proposed an interdisciplinary framework to understand this concept.

### Main problems and goals of the paper

1. **Define effective human oversight**:
- Currently, although many ethical guidelines and legislations have emphasized the importance of human oversight, the question of "when is human oversight effective" has not been fully discussed.
- The authors proposed an understanding based on risk mitigation, believing that the effectiveness of human oversight depends on four key conditions: causal power, epistemic access, self - control, and fitting intentions.

2. **Evaluate existing regulations**:
- The paper also specifically analyzed the upcoming European Union's Artificial Intelligence Act, especially the provisions regarding human oversight (such as Article 14), to test whether these regulations meet the effectiveness criteria they proposed.
- Through this analysis, the authors hope to reveal the advantages and disadvantages of existing regulations and make improvement suggestions.

### Specific explanations of the four key conditions

1. **Causal Power**:
- Definition: The supervisor must be able to exert a causal effect on the system or its impact, that is, be able to take effective measures to avoid or mitigate risks.
- For example, in a factory, the supervisor needs to have a stop button or a manual control mechanism; in an AI evaluation system, the supervisor should be able to override or ignore the system's output.

2. **Epistemic Access**:
- Definition: The supervisor must have sufficient knowledge to understand the risks and their mitigation measures.
- This includes an understanding of system functions, possible risks and benefits, intervention means and their effects. When necessary, knowledge of norms, social or cultural fields also needs to be understood.

3. **Self - Control**:
- Definition: The supervisor must be able to independently decide and execute any necessary action paths.
- This means that the supervisor, in a normal mental state and with freedom of action, can concentrate and purposefully carry out supervision work.

4. **Fitting Intentions**:
- Definition: The supervisor's intentions must match their role, that is, they should have the intention to mitigate risks and consider other relevant factors (such as the interests of users and other stakeholders).

### Conclusion

Through a detailed discussion of these four conditions, the paper aims to provide a clear conceptual framework for policymakers and researchers to better design and implement effective human oversight mechanisms, especially in high - risk AI applications. In addition, the authors also hope to promote the improvement and perfection of future regulations through their analysis.