Abstract:One of the most concrete measures to take towards meaningful AI accountability is to consequentially assess and report the systems' performance and impact. However, the practical nature of the "AI audit" ecosystem is muddled and imprecise, making it difficult to work through various concepts and map out the stakeholders involved in the practice. First, we taxonomize current AI audit practices as completed by regulators, law firms, civil society, journalism, academia, consulting agencies. Next, we assess the impact of audits done by stakeholders within each domain. We find that only a subset of AI audit studies translate to desired accountability outcomes. We thus assess and isolate practices necessary for effective AI audit results, articulating the observed connections between AI audit design, methodology and institutional context on its effectiveness as a meaningful mechanism for accountability.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve The paper aims to address several key issues in the field of Artificial Intelligence (AI) auditing: 1. **Lack of Clarity and Consistency**: The current AI auditing ecosystem is conceptually unclear, making it difficult to delineate the roles and responsibilities of various stakeholders in practice. This results in poor consistency and comparability of audit outcomes. 2. **Limited Audit Effectiveness**: Although AI auditing is widely regarded as an important means of achieving AI accountability, many audit studies fail to translate into concrete accountability outcomes. For example, few audits influence voluntary corporate actions, internal policies, product recalls, redesigns, or government policy and regulation formulation. 3. **Mismatch Between Audit Methods and Goals**: The paper points out that different types of audits (such as product audits, data audits, ecosystem audits, etc.) differ in design, methodology, and institutional context, which directly affects their effectiveness. Therefore, it is necessary to clarify which audit practices can most directly and effectively achieve audit goals. ### Solutions To address the above issues, the paper adopts the following approaches: 1. **Classifying Audit Practices**: The paper classifies current AI audit practices, including audits conducted by regulatory agencies, law firms, civil society, the media, academia, consulting firms, etc. Through this classification, the authors hope to clearly demonstrate the audit practices and their impacts in different fields. 2. **Evaluating Audit Effectiveness**: The paper evaluates the impact of audits conducted by different stakeholders and identifies which audit practices can most directly and effectively achieve audit goals. Specifically, the authors analyze the impact of audit design, methodology, and institutional context on audit effectiveness. 3. **Proposing Improvement Suggestions**: Based on the above analysis, the paper proposes suggestions for improving AI audit practices to enhance the operability and impact of audit outcomes. These suggestions aim to help auditors and policymakers better achieve accountability goals. ### Key Findings 1. **Diverse Types of Audits**: The paper categorizes audits into four main types: product/model/algorithm audits, data audits, ecosystem audits, and meta-reviews and critiques. Each type of audit differs in terms of goals, methods, and impacts. 2. **Ecosystem Audits are Most Influential**: Ecosystem audits often involve broader investigative methods such as qualitative interviews, surveys, workshops, and literature reviews. These audits not only focus on specific technical issues but also consider the impact of social and technical environments, making them more influential in driving substantive change. 3. **Audit Methods Need Diversification**: Effective audits require not only quantitative measurements but also qualitative and participatory methods. Particularly, ecosystem audits tend to adopt these methods to comprehensively assess the impact of AI systems. 4. **Audit Goals Need to be Clear**: To achieve effective accountability, audit goals must be clear and specific. This includes explicit requirements for algorithmic fairness, privacy protection, independence, and other aspects. Through these methods and findings, the paper aims to provide guidance for practitioners and policymakers in the AI auditing field to improve the effectiveness and impact of audits.

AI auditing: The Broken Bus on the Road to AI Accountability

Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing

Auditing of AI: Legal, Ethical and Technical Approaches

Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem

Artificial Intelligence in Auditing: A Conceptual Framework for Auditing Practices

From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing

Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models.

The Necessity of AI Audit Standards Boards

The algorithm audit: Scoring the algorithms that score us

Unraveling the Nuances of AI Accountability: A Synthesis of Dimensions Across Disciplines

Ethics-Based Auditing to Develop Trustworthy AI

AuditMAI: Towards An Infrastructure for Continuous AI Auditing

Audit and Assurance of AI Algorithms: A framework to ensure ethical algorithmic practices in Artificial Intelligence

A Framework for Assurance Audits of Algorithmic Systems

Accountability in AI: From principles to industry-specific accreditation

Accountability in Artificial Intelligence: What It Is and How It Works

Public vs Private Bodies: Who Should Run Advanced AI Evaluations and Audits? A Three-Step Logic Based on Case Studies of High-Risk Industries

The Dawn of the AI Robots: Towards a New Framework of AI Robot Accountability

Black-Box Access is Insufficient for Rigorous AI Audits

Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance