How do practitioners gain confidence in assurance cases?

Simon Diemert,Caleb Shortt,Jens H. Weber

2024-11-06

Abstract:CONTEXT: Assurance Cases (ACs) are prepared to argue that the system's desired quality attributes (e.g., safety or security) are satisfied. While there is strong adoption of ACs, practitioners are often left asking an important question: are we confident that the claims made by the case are true? While many confidence assessment methods (CAMs) exist, little is known about the use of these methods in practice OBJECTIVE: Develop an understanding of the current state of practice for AC confidence assessment: what methods are used in practice and what barriers exist for their use? METHOD: Structured interviews were performed with practitioners with experience contributing to real-world ACs. Open-coding was performed on transcripts. A description of the current state of AC practice and future considerations for researchers was synthesized from the results. RESULTS: A total of n = 19 practitioners were interviewed. The most common CAMs were (peer-)review of ACs, dialectic reasoning ("defeaters"), and comparing against checklists. Participants preferred qualitative methods and expressed concerns about quantitative CAMs. Barriers to using CAMs included additional work, inadequate guidance, subjectivity and interpretation of results, and trustworthiness of methods. CONCLUSION: While many CAMs are described in the literature there is a gap between the proposed methods and needs of practitioners. Researchers working in this area should consider the need to: connect CAMs to established practices, use CAMs to communicate with interest holders, crystallize the details of CAM application, curate accessible guidance, and confirm that methods are trustworthy.

Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How can practitioners have confidence in the claims in Assurance Cases (ACs)?** Specifically, the paper focuses on how practitioners evaluate and enhance their confidence in Assurance Cases in practical applications. Although many Confidence Assessment Methods (CAMs) have been proposed, little is known about the use of these methods in practice and their effectiveness. Therefore, this research aims to understand the confidence assessment methods used in current practice and the obstacles to these methods in practical applications. ### Research Background Assurance Cases (ACs) are documents used to argue that a system meets certain key quality attributes, such as safety or reliability. Although ACs are widely used in many industries, practitioners often ask an important question: Are we sure that these claims are true? ### Research Objectives The objectives of this research are to understand how practitioners evaluate confidence when preparing or managing Assurance Cases, specifically including: 1. **What are the confidence assessment methods used in current practice?** 2. **What are the obstacles to these methods in practical applications?** ### Methods The researchers conducted in - depth exchanges with experienced practitioners through structured interviews. Through open - coding analysis of the interview transcripts, the confidence assessment methods used in current practice and the existing obstacles were extracted. ### Main Findings - **Commonly used confidence assessment methods**: Peer review, dialectical reasoning ("counter - argument"), comparison with checklists, etc. - **Practitioners prefer qualitative methods**: Participants are more inclined to use qualitative methods and express concerns about quantitative methods. - **Existing obstacles**: Additional workload, lack of guidance, subjectivity of result interpretation, credibility of methods, etc. ### Conclusions Although many confidence assessment methods are described in the literature, there are obvious gaps in practical applications. The researchers suggest that future research should consider combining these methods with existing practices, providing easily accessible guidance, and ensuring the credibility of the methods. Through this research, the author hopes to provide a valuable reference for the development or improvement of confidence assessment methods in the future to better meet the practical needs of practitioners.

How do practitioners gain confidence in assurance cases?

Assessing Confidence with Assurance 2.0

How do practitioners perceive assurance cases in safety-critical software systems?

Confidence in Assurance 2.0 Cases

Security Assurance Cases -- State of the Art of an Emerging Approach

ACCESS: Assurance Case Centric Engineering of Safety-critical Systems

Automating Semantic Analysis of System Assurance Cases using Goal-directed ASP

Using Assurance Cases to Guide Verification and Validation of Research Software

A Methodology for Automating Assurance Case Generation

Defeaters and Eliminative Argumentation in Assurance 2.0

How Do Organizations Seek Cyber Assurance? Investigations on the Adoption of the Common Criteria and Beyond

Assessing safety-critical systems from operational testing: A study on autonomous vehicles

A Model for Safety Case Confidence Assessment

Assurance 2.0: A Manifesto

Towards Understanding and Applying Security Assurance Cases for Automotive Systems

Model Based System Assurance Using the Structured Assurance Case Metamodel

Identifying Conditions for Successful Advance Care Planning Implementation in Primary Care: Coincidence Analysis (RP108)

Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

I came, I saw, I certified: some perspectives on the safety assurance of cyber-physical systems

Modular Assurance of Complex Systems Using Contract-Based Design Principles