Abstract:The speed and scale at which machine learning (ML) systems are deployed are accelerating even as an increasing number of studies highlight their potential for negative impact. There is a clear need for companies and regulators to manage the risk from proposed ML systems before they harm people. To achieve this, private and public sector actors first need to identify the risks posed by a proposed ML system. A system's overall risk is influenced by its direct and indirect effects. However, existing frameworks for ML risk/impact assessment often address an abstract notion of risk or do not concretize this dependence. We propose to address this gap with a context-sensitive framework for identifying ML system risks comprising two components: a taxonomy of the first- and second-order risks posed by ML systems, and their contributing factors. First-order risks stem from aspects of the ML system, while second-order risks stem from the consequences of first-order risks. These consequences are system failures that result from design and development choices. We explore how different risks may manifest in various types of ML systems, the factors that affect each risk, and how first-order risks may lead to second-order effects when the system interacts with the real world. Throughout the paper, we show how real events and prior research fit into our Machine Learning System Risk framework (MLSR). MLSR operates on ML systems rather than technologies or domains, recognizing that a system's design, implementation, and use case all contribute to its risk. In doing so, it unifies the risks that are commonly discussed in the ethical AI community (e.g., ethical/human rights risks) with system-level risks (e.g., application, design, control risks), paving the way for holistic risk assessments of ML systems.

Safe machine learning

SAFE Artificial Intelligence in Finance

On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products

The rapid competitive economy of machine learning development: a discussion on the social risks and benefits

Engineering Safety in Machine Learning

Artificial intelligence and machine learning in dynamic cyber risk analytics at the edge

The Risks of Machine Learning Systems

Evaluating SAFE AI principles using Wasserstein distance: a comparative study of Machine Learning models

AI safety: state of the field through quantitative lens

Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges

Application for Machine Learning Methods in Financial Risk Management

Security and Safety Aspects of AI in Industry Applications

Prospects of Artificial Intelligence and Machine Learning Application in Banking Risk Management

Unsolved Problems in ML Safety

Safety design concepts for statistical machine learning components toward accordance with functional safety standards

Towards algorithms and models that we can trust: A theoretical perspective

Changing Data Sources in the Age of Machine Learning for Official Statistics

Regulatory Aspects of Artificial Intelligence and Machine Learning

How to certify machine learning based safety-critical systems? A systematic literature review

Taking Ethics, Fairness, and Bias Seriously in Machine Learning for Disaster Risk Management

Good Intentions, Risky Inventions: A Method for Assessing the Risks and Benefits of AI in Mobile and Wearable Uses