AI Procurement Checklists: Revisiting Implementation in the Age of AI Governance

Tom Zick,Mason Kortz,David Eaves,Finale Doshi-Velez

2024-04-23

Abstract:Public sector use of AI has been quietly on the rise for the past decade, but only recently have efforts to regulate it entered the cultural zeitgeist. While simple to articulate, promoting ethical and effective roll outs of AI systems in government is a notoriously elusive task. On the one hand there are hard-to-address pitfalls associated with AI-based tools, including concerns about bias towards marginalized communities, safety, and gameability. On the other, there is pressure not to make it too difficult to adopt AI, especially in the public sector which typically has fewer resources than the private sector$\unicode{x2014}$conserving scarce government resources is often the draw of using AI-based tools in the first place. These tensions create a real risk that procedures built to ensure marginalized groups are not hurt by government use of AI will, in practice, be performative and ineffective. To inform the latest wave of regulatory efforts in the United States, we look to jurisdictions with mature regulations around government AI use. We report on lessons learned by officials in Brazil, Singapore and Canada, who have collectively implemented risk categories, disclosure requirements and assessments into the way they procure AI tools. In particular, we investigate two implemented checklists: the Canadian Directive on Automated Decision-Making (CDADM) and the World Economic Forum's AI Procurement in a Box (WEF). We detail three key pitfalls around expertise, risk frameworks and transparency, that can decrease the efficacy of regulations aimed at government AI use and suggest avenues for improvement.

Computers and Society,Artificial Intelligence

What problem does this paper attempt to address?

This paper attempts to address the issue of how to ensure the effectiveness and ethics of AI systems in the process of government adoption. Specifically, the paper focuses on the following aspects:

1. **Technical assessment requires experts**: Currently, many governments have implemented detailed technical checklists to evaluate AI systems, but the effectiveness of these checklists depends on experts with relevant technical knowledge. However, expert resources are limited, which poses a challenge.

2. **Existence of procurement loopholes**: Although the procurement process touches almost all government functions, in practice, there are still loopholes in the procurement of some AI systems. For example, low-value AI procurements may not go through rigorous review processes, and nominally non-AI procurement processes may also contain AI components, thereby evading regulation.

3. **Importance of transparency**: Unlike traditional procurement goals, the impact of AI systems is highly heterogeneous, and even if they perform well in most cases, they may fail in certain specific situations. Therefore, substantive transparency (such as disclosing training data, model architecture, and performance information) and procedural transparency (such as disclosing the procurement review process) are crucial for ensuring the effectiveness and ethics of AI systems.

4. **Establishing better governance mechanisms**: To address the above challenges, the paper proposes several systematic interventions, including strengthening cooperation between the public and private sectors and academia, clarifying AI audit standards, and reasonably allocating responsibilities.

Overall, the paper aims to explore how to ensure that the government can adopt AI systems both effectively and responsibly by improving the procurement process and increasing transparency.