Abstract:To effectively address potential harms from AI systems, it is essential to identify and mitigate system-level hazards. Current analysis approaches focus on individual components of an AI system, like training data or models, in isolation, overlooking hazards from component interactions or how they are situated within a company's development process. To this end, we draw from the established field of system safety, which considers safety as an emergent property of the entire system, not just its components. In this work, we translate System Theoretic Process Analysis (STPA) - a recognized system safety framework - for analyzing AI operation and development processes. We focus on systems that rely on machine learning algorithms and conducted STPA on three case studies involving linear regression, reinforcement learning, and transformer-based generative models. Our analysis explored how STPA's control and system-theoretic perspectives apply to AI systems and whether unique AI traits - such as model opacity, capability uncertainty, and output complexity - necessitate significant modifications to the framework. We find that the key concepts and steps of conducting an STPA readily apply, albeit with a few adaptations tailored for AI systems. We present the Process-oriented Hazard Analysis for AI Systems (PHASE) as a guideline that adapts STPA concepts for AI, making STPA-based hazard analysis more accessible. PHASE enables four key affordances for analysts responsible for managing AI system harms: 1) detection of hazards at the systems level, including those from accumulation of disparate issues; 2) explicit acknowledgment of social factors contributing to experiences of algorithmic harms; 3) creation of traceable accountability chains between harms and those who can mitigate the harm; and 4) ongoing monitoring and mitigation of new hazards.
What problem does this paper attempt to address?
The core problem that this paper attempts to solve is: how to effectively identify and mitigate the potential harms that artificial intelligence (AI) systems may bring during their development and deployment. Specifically, the author points out that current harm analysis methods mainly focus on individual components of AI systems (such as training data or models), while ignoring the interactions between these components and their positions in the entire company's development process. This isolated analysis method cannot comprehensively capture system - level harms.
To solve this problem, the author introduces System - Theoretic Process Analysis (STPA) from the field of system safety. STPA is a proven system safety framework for analyzing the potential harms of complex technological systems. The author applies STPA to the operation and development processes of AI systems and explores how the control and system - theoretic perspectives of STPA are applicable to AI systems through three case studies (linear regression, reinforcement learning, and Transformer - based generative models). In addition, the author also considers whether AI - specific properties, such as model opacity, capability uncertainty, and output complexity, require significant modifications to the STPA framework.
Based on the above research, the author proposes a process - oriented hazard analysis guideline - Process - oriented Hazard Analysis for AI Systems (PHASE) to adapt to the characteristics of AI systems and make STPA - based hazard analysis more user - friendly. PHASE provides four key functions:
1. **System - level hazard detection**: including risks caused by the accumulation of different problems.
2. **Clarifying the influence of social factors**: acknowledging the influence of social factors on the experience of algorithmic harms.
3. **Establishing a traceable chain of responsibility**: creating a chain of responsibility from hazards to those responsible for mitigation.
4. **Continuously monitoring and mitigating new hazards**: ensuring that new risks can be detected and dealt with in a timely manner.
### Research Background and Methods
#### Background
The author first reviews the history and development of system safety engineering, emphasizing the importance of system safety frameworks in analyzing complex systems. Then, they introduce the basic steps of STPA, including:
- **Determining the purpose of analysis**: identifying losses and hazards.
- **Creating a control diagram**: visualizing the system and the elements and interactions within its boundaries.
- **Identifying unsafe control behaviors**: finding control behaviors that may lead to hazards.
- **Identifying loss scenarios**: analyzing situations that may lead to unsafe control behaviors.
#### Methods
To verify the applicability of STPA in AI systems, the author selects three representative case studies:
1. **Early warning system**: using the linear regression algorithm to predict the probability of late - onset sepsis in premature infants.
2. **Insulin injection system**: a personalized automatic insulin injection system based on reinforcement learning.
3. **Storyboard creation**: a creative practice based on a text - to - image generative model.
Through the study of these three cases, the author gradually explains how to apply STPA to AI systems and finally forms the PHASE guideline.
### Conclusion
The main contributions of this research are:
- Proposing a STPA - based guideline (PHASE) applicable to AI systems.
- Demonstrating the application effect of STPA through three diverse cases.
- Providing four key functions to help more comprehensively analyze and manage the potential harms of AI systems.
Through this method, the author hopes to promote the safety assessment of AI systems from the component level to the system level, so as to better cope with the complex challenges brought by AI systems.