Abstract:The use of AI technologies is percolating into the secure development of software-based systems, with an increasing trend of composing AI-based subsystems (with uncertain levels of performance) into automated pipelines. This presents a fundamental research challenge and poses a serious threat to safety-critical domains (e.g., aviation). Despite the existing knowledge about uncertainty in risk analysis, no previous work has estimated the uncertainty of AI-augmented systems given the propagation of errors in the pipeline. We provide the formal underpinnings for capturing uncertainty propagation, develop a simulator to quantify uncertainty, and evaluate the simulation of propagating errors with two case studies. We discuss the generalizability of our approach and present policy implications and recommendations for aviation. Future work includes extending the approach and investigating the required metrics for validation in the aviation domain.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in an AI - enhanced security pipeline, how to quantify and manage the impact of uncertainty propagation on system performance, especially in applications in safety - critical areas (such as aviation). Specifically, the author focuses on how error propagation of AI components affects the overall system performance evaluation in an automated security pipeline. ### Problem Background With the wide application of artificial intelligence (AI) technology in software development, more and more AI subsystems are integrated into automated pipelines. The performance of these AI subsystems is uncertain, which poses a serious threat to safety - critical areas (such as aviation). Although existing research has explored uncertainty in risk analysis, no work has yet estimated the uncertainty due to error propagation within the pipeline in AI - enhanced systems. ### Research Challenges The main research challenges are: 1. **Uncertainty Propagation**: Each component in an AI - enhanced system can be a potential source of error, and these errors will propagate and affect the performance evaluation of the final system. 2. **Lack of Global Risk Metric**: Existing AI - enhanced systems usually only provide point estimates or completely ignore uncertainty when evaluating risks, which makes it difficult for decision - makers to understand the inherent uncertainty in model predictions. ### Research Objectives The author proposes the following research questions: - **RQ**: How to estimate the total error (or success rate) of an AI - enhanced system, taking into account the propagation error of classifiers in the pipeline? ### Solutions To solve these problems, the author provides the following contributions: 1. **Formalization Basis**: Established a formalization basis for capturing uncertainty propagation in an AI - enhanced automatic program repair (APR) pipeline. 2. **Simulator Development**: Developed a simulator to quantify the impact of uncertainty propagation on automated APR tools. 3. **Case Studies**: Through two case studies, calculated the impact of uncertainty on the two proposed solutions. 4. **Policy Recommendations**: Discussed policy implications and proposed recommended measures for the aviation field. ### Key Observations 1. **Uncertainty Quantification**: Although uncertainty quantification methods in the AI field are quite mature, these methods mainly focus on a single model and do not consider uncertainty propagation between multiple models. 2. **Vulnerability Detection and Repair**: Although there are many studies on vulnerability detection and automatic repair, these studies usually do not quantify the uncertainty of their methods, which may lead to overestimation of system performance. 3. **Error Propagation in Automated Pipelines**: Recent research shows that error propagation in automated vulnerability repair pipelines has not been fully studied, which poses a challenge to the overall performance evaluation of the system. ### Conclusion This paper aims to provide a more accurate risk assessment method for AI - enhanced security pipelines by establishing a theoretical basis for uncertainty propagation and developing corresponding simulation tools, thereby improving the safety and reliability in safety - critical areas (such as aviation).

Risks of uncertainty propagation in Al-augmented security pipelines

System-Level Analysis of Module Uncertainty Quantification in the Autonomy Pipeline

Uncertainty Aware AI ML: Why and How

The Reasoning Under Uncertainty Trap: A Structural AI Risk

Uncertainty-aware Risk Assessment of Robotic Systems via Importance Sampling

Inherently Interpretable and Uncertainty-Aware Models for Online Learning in Cyber-Security Problems

Propagation and Attribution of Uncertainty in Medical Imaging Pipelines

Evaluation of Uncertainty Quantification in Deep Learning

An Exploratory Study of AI System Risk Assessment from the Lens of Data Distribution and Uncertainty

Uncertainties in Onboard Algorithms for Autonomous Vehicles: Challenges, Mitigation, and Perspectives.

Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification

Scope Compliance Uncertainty Estimate

A Proposed Artificial Intelligence Workflow to Address Application Challenges Leveraged on Algorithm Uncertainty

A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation

Speculations on Uncertainty and Humane Algorithms

Towards an Ontology-Driven Approach for Process-Aware Risk Propagation

Examining the Differential Risk from High-level Artificial Intelligence and the Question of Control

Inherent Diverse Redundant Safety Mechanisms for AI-based Software Elements in Automotive Applications

Security-Informed Safety Analysis of Autonomous Transport Systems Considering AI-Powered Cyberattacks and Protection

Quantitative AI Risk Assessments: Opportunities and Challenges

Interpretable Uncertainty Quantification in AI for HEP