Abstract:We outline the principles of classical assurance for computer-based systems that pose significant risks. We then consider application of these principles to systems that employ Artificial Intelligence (AI) and Machine Learning (ML). A key element in this "dependability" perspective is a requirement to have near-complete understanding of the behavior of critical components, and this is considered infeasible for AI and ML. Hence the dependability perspective aims to minimize trust in AI and ML elements by using "defense in depth" with a hierarchy of less complex systems, some of which may be highly assured conventionally engineered components, to "guard" them. This may be contrasted with the "trustworthy" perspective that seeks to apply assurance to the AI and ML elements themselves. In cyber-physical and many other systems, it is difficult to provide guards that do not depend on AI and ML to perceive their environment (e.g., other vehicles sharing the road with a self-driving car), so both perspectives are needed and there is a continuum or spectrum between them. We focus on architectures toward the dependability end of the continuum and invite others to consider additional points along the spectrum. For guards that require perception using AI and ML, we examine ways to minimize the trust placed in these elements; they include diversity, defense in depth, explanations, and micro-ODDs. We also examine methods to enforce acceptable behavior, given a model of the world. These include classical cyber-physical calculations and envelopes, and normative rules based on overarching principles, constitutions, ethics, or reputation. We apply our perspective to autonomous systems, AI systems for specific functions, generic AI such as Large Language Models, and to Artificial General Intelligence (AGI), and we propose current best practice and an agenda for research.

Revising human-systems engineering principles for embedded AI applications

Requirement Engineering Challenges for AI-intense Systems Development

Testing Artificial Intelligence System Towards Safety and Robustness: State of the Art

Human-centred test and evaluation of military AI

Inherent Diverse Redundant Safety Mechanisms for AI-based Software Elements in Automotive Applications

Challenges of Human-Aware AI Systems

How Mature is Requirements Engineering for AI-based Systems? A Systematic Mapping Study on Practices, Challenges, and Future Research Directions

Superintelligence Safety: A Requirements Engineering Perspective

Assurance of AI Systems From a Dependability Perspective

A Case for Competent AI Systems $-$ A Concept Note

Exploring Opportunities in Usable Hazard Analysis Processes for AI Engineering

Reliability, Resilience and Human Factors Engineering for Trustworthy AI Systems

An Integrative and Transdisciplinary Approach for a Human-Centered Design of AI-Based Work Systems

Requirements Engineering Framework for Human-centered Artificial Intelligence Software Systems

AI‐Augmented Software Engineering: Revolutionizing or Challenging Software Quality and Testing?

Requirements Elicitation and Modelling of Artificial Intelligence Systems: An Empirical Study

Some aspects of software engineering for AI-based systems

Human-AI Safety: A Descendant of Generative AI and Control Systems Safety

Engineering Reliable Deep Learning Systems

What's my role? Modelling responsibility for AI-based safety-critical systems