Abstract:Many stakeholders struggle to make reliances on ML-driven systems due to the risk of harm these systems may cause. Concerns of trustworthiness, unintended social harms, and unacceptable social and ethical violations undermine the promise of ML advancements. Moreover, such risks in complex ML-driven systems present a special challenge as they are often difficult to foresee, arising over periods of time, across populations, and at scale. These risks often arise not from poor ML development decisions or low performance directly but rather emerge through the interactions amongst ML development choices, the context of model use, environmental factors, and the effects of a model on its target. Systems safety engineering is an established discipline with a proven track record of identifying and managing risks even in high-complexity sociotechnical systems. In this work, we apply a state-of-the-art systems safety approach to concrete applications of ML with notable social and ethical risks to demonstrate a systematic means for meeting the assurance requirements needed to argue for safe and trustworthy ML in sociotechnical systems.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to ensure safety and trustworthiness in complex machine - learning (ML) - driven systems, especially in applications involving social and ethical risks**. Specifically, the authors are concerned that current methods for evaluating and managing risks in ML systems mainly focus on the component level and overlook the social and ethical impacts at the system level. This limitation has led to insufficient risk assessment on a large - scale, long - term, and across - population basis. ### Core Problems of the Paper 1. **Limitations of Existing Methods**: - Most current interventions either focus on improving specific performance indicators or optimizing specific aspects of ML development practices, or are management and policy frameworks, such as NIST's AI Risk Management Framework. - These methods are often difficult to translate from high - level policy goals and ethical ideals into implementable technical decisions. 2. **Risks at the System Level**: - The complexity of social and technical systems makes risks difficult to foresee, especially under the influence of interactions, environmental factors, and the context of model use. - Improvements at the component level cannot ensure the safety of the entire system because system - level risks are often caused by the interactions between components. 3. **Trust and Social - Ethical Issues**: - The trustworthiness of ML systems, unexpected social harms, and unacceptable social and ethical violations undermine the potential of ML technology. - Existing methods are difficult to evaluate and manage these social and ethical risks at the system level. ### Solutions To solve the above problems, the authors introduced the method of **Systems Safety Engineering (SSE)** and applied it to specific ML system case studies. The specific steps are as follows: 1. **Apply STPA (System - Theoretic Process Analysis)**: - STPA is a top - down system safety analysis method that has been successful in high - consequence fields such as nuclear power plants and spaceflight. - By defining the analysis purpose, modeling the system control structure, identifying Unsafe Control Actions (UCAs), and considering loss scenarios, STPA can effectively identify and manage social and ethical risks. 2. **Case Studies**: - **Prescription Drug Monitoring Programs (PDMP)**: Analyzed the ML system used to predict the risk of drug abuse and explored its impact on patients and society. - **Criminal Justice Facial Recognition Systems (CJFR)**: Analyzed the application of facial recognition technology in law enforcement, especially its potential social and ethical risks. ### Main Findings - **PDMP Case**: Revealed the gap between the system optimization goals and actual social needs, such as the relationship between reducing drug abuse and preventing overdose deaths. - **CJFR Case**: Emphasized the wide application of facial recognition systems in law enforcement and their lack of standardized guidance and governance mechanisms, which may lead to misjudgments and social unfairness. Through these analyses, the authors demonstrated how to use SSE tools and techniques to systematically identify and manage social and ethical risks in ML systems, thereby providing guarantees for safe and trustworthy ML systems.

Concrete Safety for ML Problems: System Safety for ML Development and Assessment

System Safety Engineering for Social and Ethical ML Risks: A Case Study

Unsolved Problems in ML Safety

From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML

On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products

Beyond the ML Model: Applying Safety Engineering Frameworks to Text-to-Image Development

Concrete Problems in AI Safety, Revisited

Engineering Safety in Machine Learning

The Risks of Machine Learning Systems

How to certify machine learning based safety-critical systems? A systematic literature review

Safety Cases: How to Justify the Safety of Advanced AI Systems

Taxonomy of Machine Learning Safety: A Survey and Primer

Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety

Safety design concepts for statistical machine learning components toward accordance with functional safety standards

Toward Sociotechnical AI: Mapping Vulnerabilities for Machine Learning in Context

The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

Safeguarding AI Agents: Developing and Analyzing Safety Architectures

Systematically Assessing the Security Risks of AI/ML-enabled Connected Healthcare Systems

SoK: Machine Learning Governance

A Holistic Assessment of the Reliability of Machine Learning Systems

Towards the Science of Security and Privacy in Machine Learning