Abstract:Human oversight has become a key mechanism for the governance of artificial intelligence (“AI”). Human overseers are supposed to increase the accuracy and safety of AI systems, uphold human values in automated decision-making, and build trust in the technology. Empirical research suggests, however, that humans are not reliable in fulfilling their oversight tasks. This article shows how the challenge of trustworthy and meaningful human oversight of AI can productively be addressed through the lens of normative institutional design. It draws on scholarship within democratic theory and suggests that the idea of “institutionalised distrust” provides a scaffolding along which local implementations of human oversight of AI systems may be built. Applied to AI governance, institutionalising distrust means utilising reasonable distrust in the abilities and motivations of human overseers for the design of their roles. The article aims to make three contributions. First, it presents a simplified taxonomy of human oversight roles, differentiated along their degree of influence on the decisional output of an AI system. Second, it connects the resulting two types of oversight roles with two institutional challenges for their successful implementation: a lack of competence and harmful incentives. Third, to address these challenges, the article formulates six normative design principles which institutionalise distrust in human oversight of AI.

Institutionalised Distrust and Human Oversight of Artificial Intelligence: Toward a Democratic Design of AI Governance under the European Union AI Act