Abstract:Context: Demonstrating high reliability and safety for safety-critical systems (SCSs) remains a hard problem. Diverse evidence needs to be combined in a rigorous way: in particular, results of operational testing with other evidence from design and verification. Growing use of machine learning in SCSs, by precluding most established methods for gaining assurance, makes evidence from operational testing even more important for supporting safety and reliability claims.Objective: We revisit the problem of using operational testing to demonstrate high reliability. We use Autonomous Vehicles (AVs) as a current example. AVs are making their debut on public roads: methods for assessing whether an AV is safe enough are urgently needed. We demonstrate how to answer 5 questions that would arise in assessing an AV type, starting with those proposed by a highly-cited study.Method: We apply new theorems extending our Conservative Bayesian Inference (CBI) approach, which exploit the rigour of Bayesian methods while reducing the risk of involuntary misuse associated (we argue) with now-common applications of Bayesian inference; we define additional conditions needed for applying these methods to AVs.Results: Prior knowledge can bring substantial advantages if the AV design allows strong expectations of safety before road testing. We also show how naive attempts at conservative assessment may lead to over-optimism instead; why extrapolating the trend of disengagements (take-overs by human drivers) is not suitable for safety claims; use of knowledge that an AV has moved to a "less stressful" environment.Conclusion: While some reliability targets will remain too high to be practically verifiable, our CBI approach removes a major source of doubt: it allows use of prior knowledge without inducing dangerously optimistic biases. For certain ranges of required reliability and prior beliefs, CBI thus supports feasible, sound arguments. Useful conservative claims can be derived from limited prior knowledge.

Road to safe autonomy with data and formal reasoning

Online Safety Verification of Autonomous Driving Decision-Making Based on Dynamic Reachability Analysis

Negotiation-Aware Reachability-Based Safety Verification for AutonomousDriving in Interactive Scenarios

Self-driving car safety quantification via component-level analysis

Verifying Collision Risk Estimation using Autonomous Driving Scenarios Derived from a Formal Model

Integrated Modular Safety System Design for Intelligent Autonomous Vehicles

Slow Down, Move Over: A Case Study in Formal Verification, Refinement, and Testing of the Responsibility-Sensitive Safety Model for Self-Driving Cars

Hybrid Verification Technique for Decision-Making of Self-Driving Vehicles

Formal Development of Safe Automated Driving using Differential Dynamic Logic

A Formally Verified Fail-Operational Safety Concept for Automated Driving

Formal Certification Methods for Automated Vehicle Safety Assessment

Safe by Design Autonomous Driving Systems

Assessing safety-critical systems from operational testing: A study on autonomous vehicles

Formal Verification of Safety Architectures for Automated Driving

Learning Autonomous Vehicle Safety Concepts from Demonstrations

Self-Driving Vehicle Verification Towards a Benchmark

A formal framework for the safe design of the Autonomous Driving supervision

Verifying Controllers with Convolutional Neural Network-based Perception: A Case for Intelligible, Safe, and Precise Abstractions

DRYVR:Data-driven verification and compositional reasoning for automotive systems

Safety and Risk Analysis of Autonomous Vehicles Using Computer Vision and Neural Networks

Model-based Verification and Validation of an Autonomous Vehicle System