Abstract:Multi-agent cyber-physical systems are present in a variety of applications. Agent decision-making can be affected due to errors induced by uncertain, dynamic operating environments or due to incorrect actions taken by an agent. When an erroneous decision that leads to a violation of safety is identified, assigning responsibility to individual agents is a key step toward preventing future accidents. Current approaches to carrying out such investigations require human labor or high degree of familiarity with operating environments. Automated strategies to assign responsibility can achieve a significant reduction in human effort and associated cognitive burden. In this paper, we develop an automated procedure to assign responsibility for safety violations to actions of any single agent in a principled manner. We base our approach on reasoning about safety violations in road safety. Given a safety violation, we use counterfactual reasoning to create alternative scenarios, showing how different outcomes could have occurred if certain actions had been replaced by others. We introduce the degree of responsibility (DoR) metric for each agent. The DoR, using the Shapley value, quantifies each agent's contribution to the safety violation, providing a basis to explain and justify decisions. We also develop heuristic techniques and methods based on agent interaction structures to improve scalability as agent numbers grow. We examine three safety violation cases from the National Highway Traffic Safety Administration (NHTSA). We run experiments using CARLA urban driving simulator. Results show the DoR improves the explainability of decisions and accountability for agent actions and their consequences.

Responsibility Modeling for the Sociotechnical Risk Analysis of Coalitions of Systems

An asymmetric interdependent networks model for cyber-physical systems

What's my role? Modelling responsibility for AI-based safety-critical systems

Designing Accountable Systems

Concrete Safety for ML Problems: System Safety for ML Development and Assessment

On Heuristic Models, Assumptions, and Parameters

Development of Rasmussen's risk management framework for analysing multi-level sociotechnical influences in the design of envisioned work systems

Expressing Accountability Patterns using Structural Causal Models

Dynamic Risk Management in Cyber Physical Systems

Quantifying Responsibility with Probabilistic Causation -- The Case of Climate Action

Complex systems perspective in assessing risks in artificial intelligence

An ontological metamodel for cyber-physical system safety, security, and resilience coengineering

Responsibility in Actor-Based Systems

Moving Beyond Normal Accidents and High Reliability Organizations: A Systems Approach to Safety in Complex Systems

The Responsibility Quantification (ResQu) Model of Human Interaction with Automation

Hazard Analysis of Collaborative Automation Systems: A Two-layer Approach based on Supervisory Control and Simulation

Who is Responsible? Explaining Safety Violations in Multi-Agent Cyber-Physical Systems

Applying the "SOTEC" framework of sociotechnical risk analysis to the development of an autonomous robot swarm for a public cloakroom

Fundamental Challenges of Cyber-Physical Systems Security Modeling

Risk Structures: Towards Engineering Risk-aware Autonomous Systems

Cyber Defense as a Complex Adaptive System: A model-based approach to strategic policy design