Public Procurement for Responsible AI? Understanding U.S. Cities' Practices, Challenges, and Needs

Nari Johnson,Elise Silva,Harrison Leon,Motahhare Eslami,Beth Schwanke,Ravit Dotan,Hoda Heidari

2024-11-08

Abstract:Most AI tools adopted by governments are not developed internally, but instead are acquired from third-party vendors in a process called public procurement. While scholars and regulatory proposals have recently turned towards procurement as a site of intervention to encourage responsible AI governance practices, little is known about the practices and needs of city employees in charge of AI procurement. In this paper, we present findings from semi-structured interviews with 18 city employees across 7 US cities. We find that AI acquired by cities often does not go through a conventional public procurement process, posing challenges to oversight and governance. We identify five key types of challenges to leveraging procurement for responsible AI that city employees face when interacting with colleagues, AI vendors, and members of the public. We conclude by discussing recommendations and implications for governments, researchers, and policymakers.

Computers and Society,Artificial Intelligence,Human-Computer Interaction

What problem does this paper attempt to address?

The problems that this paper attempts to solve are: understanding the practices, challenges, and needs that American cities face when procuring responsible artificial intelligence (AI) systems. Specifically, the research aims to answer two key questions:

1. **What are the existing public - sector AI procurement practices?**
- What practices do city employees follow when obtaining AI products and services?

2. **What challenges and needs do city employees face when procuring AI responsibly?**
- What challenges do they encounter and what resources are required to overcome these challenges?

### Background and Motivation

With the wide application of artificial intelligence technology in the public sector, the government usually purchases AI tools from third - party suppliers through the "public procurement" process rather than developing them in - house. However, academia and regulatory advice have only recently begun to focus on how to encourage responsible AI governance practices through the procurement process. Nevertheless, little is known about the specific practices and needs of city employees responsible for AI procurement.

### Research Methods

To fill this knowledge gap, the authors conducted semi - structured interviews with 18 city employees from 7 American cities. These interviews revealed the following:

- **Existing AI procurement practices**: All of the surveyed cities have begun to use externally - developed AI technologies for various purposes, such as assisting law enforcement, urban planning decision - making, administrative decision - support, resident communication, and improving work efficiency. Most cities do not have the ability to develop AI solutions in - house and therefore rely on external suppliers.

- **Challenges in the procurement process**: Many AI product procurements do not go through the traditional procurement process but are obtained through alternative routes, which poses challenges for supervision and governance. For example, many AI products are below the cost threshold for competitive bidding, reflecting the wide availability of low - cost commercial AI tools.

### Main Findings

1. **Many AI procurements occur outside the traditional procurement process**:
- In the surveyed cities, two - thirds of the cities have changed their procurement practices to meet the needs of AI procurement, such as introducing AI - specific RFP clauses or contract clauses, or establishing IT - led AI review processes. The remaining four cities have not yet changed their procurement practices.

2. **City organizations face five major categories of challenges in AI procurement**:
- **Lack of visibility**: It is difficult to understand the use of AI by employees and suppliers, which is a prerequisite for supervision and governance.
- **Organizational barriers**: Time constraints for prioritizing AI governance work and opposition from other colleagues.
- **Insufficient preparation**: Inadequate ability to assess AI risks and difficulty in interpreting the quantitative performance indicators provided by suppliers.
- **Weak relationship with suppliers**: Suppliers refuse to answer basic questions on the grounds of trade secrets, do not involve the city in AI design, and shift the review and supervision responsibilities to the city.
- **Stakeholder participation**: Involves the tension between public supervision and legal confidentiality obligations.

### Conclusions and Recommendations

Based on these findings, the authors put forward suggestions for governments, researchers, and policymakers to better support local governments in effectively using public procurement as an intervention point to promote responsible AI applications.