Abstract:To effectively address potential harms from AI systems, it is essential to identify and mitigate system-level hazards. Current analysis approaches focus on individual components of an AI system, like training data or models, in isolation, overlooking hazards from component interactions or how they are situated within a company's development process. To this end, we draw from the established field of system safety, which considers safety as an emergent property of the entire system, not just its components. In this work, we translate System Theoretic Process Analysis (STPA) - a recognized system safety framework - for analyzing AI operation and development processes. We focus on systems that rely on machine learning algorithms and conducted STPA on three case studies involving linear regression, reinforcement learning, and transformer-based generative models. Our analysis explored how STPA's control and system-theoretic perspectives apply to AI systems and whether unique AI traits - such as model opacity, capability uncertainty, and output complexity - necessitate significant modifications to the framework. We find that the key concepts and steps of conducting an STPA readily apply, albeit with a few adaptations tailored for AI systems. We present the Process-oriented Hazard Analysis for AI Systems (PHASE) as a guideline that adapts STPA concepts for AI, making STPA-based hazard analysis more accessible. PHASE enables four key affordances for analysts responsible for managing AI system harms: 1) detection of hazards at the systems level, including those from accumulation of disparate issues; 2) explicit acknowledgment of social factors contributing to experiences of algorithmic harms; 3) creation of traceable accountability chains between harms and those who can mitigate the harm; and 4) ongoing monitoring and mitigation of new hazards.

A Taxonomy for AI Hazard Analysis

A taxonomic system for failure cause analysis of open source AI incidents

AI Hazard Management: A framework for the systematic management of root causes for AI risks

Exploring Opportunities in Usable Hazard Analysis Processes for AI Engineering

From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems

Enhancing Autonomous Driving Safety Analysis with Generative AI: A Comparative Study on Automated Hazard and Risk Assessment

An Overview of Catastrophic AI Risks

A Taxonomy of Systemic Risks from General-Purpose AI

Explainable artificial intelligence in transport Logistics: Risk analysis for road accidents

Understanding and Avoiding AI Failures: A Practical Guide

The AI Risk Repository: A Comprehensive Meta-Review, Database, and Taxonomy of Risks From Artificial Intelligence

The Digital Divide in Process Safety: Quantitative Risk Analysis of Human-AI Collaboration

Examining the Differential Risk from High-level Artificial Intelligence and the Question of Control

Considerations Influencing Offense-Defense Dynamics From Artificial Intelligence

Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review

Artificial Intelligence for Safety-Critical Systems in Industrial and Transportation Domains: A Survey

On Safety Assessment of Artificial Intelligence

Artificial intelligence for risk analysis-A risk characterization perspective on advances, opportunities, and limitations

Artificial intelligence (AI) and process safety: Some cautionary observations

Safety Cases: How to Justify the Safety of Advanced AI Systems

Quantitative AI Risk Assessments: Opportunities and Challenges