Abstract:AI auditing is a rapidly growing field of research and practice. This review article, which doubles as an editorial to Digital Societys topical collection on Auditing of AI, provides an overview of previous work in the field. Three key points emerge from the review. First, contemporary attempts to audit AI systems have much to learn from how audits have historically been structured and conducted in areas like financial accounting, safety engineering and the social sciences. Second, both policymakers and technology providers have an interest in promoting auditing as an AI governance mechanism. Academic researchers can thus fill an important role by studying the feasibility and effectiveness of different AI auditing procedures. Third, AI auditing is an inherently multidisciplinary undertaking, to which substantial contributions have been made by computer scientists and engineers as well as social scientists, philosophers, legal scholars and industry practitioners. Reflecting this diversity of perspectives, different approaches to AI auditing have different affordances and constraints. Specifically, a distinction can be made between technology-oriented audits, which focus on the properties and capabilities of AI systems, and process oriented audits, which focus on technology providers governance structures and quality management systems. The next step in the evolution of auditing as an AI governance mechanism, this article concludes, should be the interlinking of these available (and complementary) approaches into structured and holistic procedures to audit not only how AI systems are designed and used but also how they impact users, societies and the natural environment in applied settings over time.

What problem does this paper attempt to address?

The main issue this paper attempts to address is the methods, importance, and development of auditing artificial intelligence systems (AI systems) at legal, ethical, and technical levels. Specifically: 1. **Definition and Methodology**: - The paper explores how to define AI auditing from functional and methodological perspectives. Functionally, AI auditing is a governance mechanism; methodologically, it refers to a process of evaluating whether an entity's behavior conforms to predefined standards. 2. **Historical References**: - The paper reviews the historical development of auditing in financial accounting, safety engineering, and social science research to draw lessons from these fields. For example, financial auditing emphasizes the importance of transparency and independence, while safety auditing focuses on the comprehensive assessment of both technical and organizational culture. 3. **Current Needs**: - The study points out that the modern society's need for AI system auditing stems from two pressures: top-down policy pushes (such as government regulatory requirements) and bottom-up corporate initiatives (to identify and manage risks). This indicates that both policymakers and technology providers are interested in using auditing as an effective AI governance tool. 4. **Comparison of Different Methods**: - The paper distinguishes between narrow (technology-oriented) and broad (process-oriented) AI auditing methods, noting that each has its pros and cons. The former focuses on evaluating AI output results, while the latter pays more attention to the quality management systems of technology providers. 5. **Future Directions**: - Finally, the authors suggest that future AI auditing should combine existing tools and methods to form a structured and comprehensive procedure. This should not only review the design and usage of AI but also consider its impact on users, society, and the natural environment in practical applications. In summary, this paper aims to propose a more systematic auditing framework through a review of the AI auditing field, to better manage and mitigate the various risks that AI technology may bring.

Auditing of AI: Legal, Ethical and Technical Approaches

Artificial Intelligence in Auditing: A Conceptual Framework for Auditing Practices

Ethics-Based Auditing to Develop Trustworthy AI

From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing

Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models.

Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem

AI in Auditing: A Comprehensive Review of Applications, Benefits and Challenges

Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

AI auditing: The Broken Bus on the Road to AI Accountability

Towards algorithm auditing: managing legal, ethical and technological risks of AI, ML and associated algorithms

Advancing AI Audits for Enhanced AI Governance

The Necessity of AI Audit Standards Boards

Public vs Private Bodies: Who Should Run Advanced AI Evaluations and Audits? A Three-Step Logic Based on Case Studies of High-Risk Industries

Audit and Assurance of AI Algorithms: A framework to ensure ethical algorithmic practices in Artificial Intelligence

The Ethical Implications of Using Artificial Intelligence in Auditing

A Framework for Assurance Audits of Algorithmic Systems

AuditMAI: Towards An Infrastructure for Continuous AI Auditing

Operationalising AI governance through ethics-based auditing: An industry case study

Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing

A Blueprint for Auditing Generative AI

Designing an ML Auditing Criteria Catalog as Starting Point for the Development of a Framework