The Necessity of AI Audit Standards Boards

David Manheim,Sammy Martin,Mark Bailey,Mikhail Samin,Ross Greutzmacher

2024-04-11

Abstract:Auditing of AI systems is a promising way to understand and manage ethical problems and societal risks associated with contemporary AI systems, as well as some anticipated future risks. Efforts to develop standards for auditing Artificial Intelligence (AI) systems have therefore understandably gained momentum. However, we argue that creating auditing standards is not just insufficient, but actively harmful by proliferating unheeded and inconsistent standards, especially in light of the rapid evolution and ethical and safety challenges of AI. Instead, the paper proposes the establishment of an AI Audit Standards Board, responsible for developing and updating auditing methods and standards in line with the evolving nature of AI technologies. Such a body would ensure that auditing practices remain relevant, robust, and responsive to the rapid advancements in AI. The paper argues that such a governance structure would also be helpful for maintaining public trust in AI and for promoting a culture of safety and ethical responsibility within the AI industry.

Computers and Society,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the deficiencies of current artificial intelligence (AI) audit standards and their potential harms. Specifically, the paper points out:

1. **Limitations of Static Standards**: Most of the existing AI audit standards are static and cannot keep up with the rapid evolution of AI technology. These standards may be reasonable when formulated, but with the progress of technology, they will soon become obsolete and cannot effectively address new ethical and safety challenges.

2. **Inconsistent and Fragmented Standards**: Due to the lack of a unified coordinating body, different organizations and countries have developed their own standards, resulting in conflicts and duplications among the standards. This fragmentation makes the standards difficult to implement and reduces their effectiveness.

3. **Ignoring Audits of the Entire Development Process**: Current audits mainly focus on the final product and ignore the entire development process of the AI system. However, the risks of the AI system not only come from the final product but may also originate from various stages in the development process, such as data collection, model training, etc.

4. **Complexity of Ethical and Social Risks**: AI systems not only involve technical issues but also bring complex ethical and social risks. These risks need to be jointly evaluated and managed through the participation of multiple stakeholders, rather than relying solely on technical evaluations.

To solve these problems, the paper proposes to establish a special **AI Audit Standards Board** (AI Audit Standards Board), which is responsible for formulating and updating audit methods and standards that are compatible with the development of AI technology. This committee will ensure that audit practices can keep up with the rapid development of AI technology, maintain relevance and robustness, and at the same time promote public trust in AI and promote the formation of a culture of safety and ethical responsibility in the AI industry.

### Formula Representation

Although this paper does not involve specific mathematical formulas, when discussing the iterative development and evaluation of AI models, some common machine - learning formulas can be introduced to illustrate the performance evaluation and risk analysis of the models. For example, for the risk assessment of the model, the following formula can be used to represent it:

\[ R = E[L(y, \hat{y})] \]

where:
- \( R \) represents the risk of the model.
- \( E \) represents the expected value.
- \( L(y, \hat{y}) \) represents the loss function, \( y \) is the true label, and \(\hat{y}\) is the model - predicted value.

In addition, when discussing the bias and variance of the model, the following formulas can be used:

\[ \text{Bias}^2=(E[\hat{f}(x)] - f(x))^2 \]
\[ \text{Variance}=E[(\hat{f}(x)-E[\hat{f}(x)])^2] \]

These formulas can help understand various aspects of model performance, so as to better conduct audits and risk assessments.

### Summary

The core problem of the paper is the deficiencies of current AI audit standards and their potential harms, and it proposes to solve these problems by establishing an AI audit standards committee. This will help ensure that audit standards can keep up with the rapid development of AI technology, maintain relevance and robustness, and at the same time promote public trust in AI and the culture of safety and ethical responsibility within the AI industry.