Abstract:Context: Demonstrating high reliability and safety for safety-critical systems (SCSs) remains a hard problem. Diverse evidence needs to be combined in a rigorous way: in particular, results of operational testing with other evidence from design and verification. Growing use of machine learning in SCSs, by precluding most established methods for gaining assurance, makes evidence from operational testing even more important for supporting safety and reliability claims.Objective: We revisit the problem of using operational testing to demonstrate high reliability. We use Autonomous Vehicles (AVs) as a current example. AVs are making their debut on public roads: methods for assessing whether an AV is safe enough are urgently needed. We demonstrate how to answer 5 questions that would arise in assessing an AV type, starting with those proposed by a highly-cited study.Method: We apply new theorems extending our Conservative Bayesian Inference (CBI) approach, which exploit the rigour of Bayesian methods while reducing the risk of involuntary misuse associated (we argue) with now-common applications of Bayesian inference; we define additional conditions needed for applying these methods to AVs.Results: Prior knowledge can bring substantial advantages if the AV design allows strong expectations of safety before road testing. We also show how naive attempts at conservative assessment may lead to over-optimism instead; why extrapolating the trend of disengagements (take-overs by human drivers) is not suitable for safety claims; use of knowledge that an AV has moved to a "less stressful" environment.Conclusion: While some reliability targets will remain too high to be practically verifiable, our CBI approach removes a major source of doubt: it allows use of prior knowledge without inducing dangerously optimistic biases. For certain ranges of required reliability and prior beliefs, CBI thus supports feasible, sound arguments. Useful conservative claims can be derived from limited prior knowledge.

Challenges of Assured Autonomy

Grand Challenges in the Verification of Autonomous Systems

Assurance for Autonomy -- JPL's past research, lessons learned, and future directions

Ensuring Safe Autonomy: Navigating the Future of Autonomous Vehicles

Assurance of AI Systems From a Dependability Perspective

EDA for autonomous behavior assurance

"I can assure you [$\ldots$] that it's going to be all right" -- A definition, case for, and survey of algorithmic assurances in human-autonomy trust relationships

Verification and Validation of Autonomous Systems

A Review of Theoretical and Practical Challenges of Trusted Autonomy in Big Data

Assured Autonomy: Path Toward Living With Autonomous Systems We Can Trust

Formal Specification and Verification of Autonomous Robotic Systems: A Survey

Road to safe autonomy with data and formal reasoning

Improving Competence for Reliable Autonomy

Towards Automated Safety Coverage and Testing for Autonomous Vehicles with Reinforcement Learning

Algorithmic Ethics: Formalization and Verification of Autonomous Vehicle Obligations

Open Challenges in the Formal Verification of Autonomous Driving

Autonomous Vehicle Safety: An Interdisciplinary Challenge

A Unified Approach to 3 Basic Challenges in Shared Autonomy

Security Challenges in Autonomous Systems Design

Assessing safety-critical systems from operational testing: A study on autonomous vehicles

Compositional Verification for Autonomous Systems with Deep Learning Components