Grand Challenges in the Verification of Autonomous Systems

Kevin Leahy,Hamid Asgari,Louise A. Dennis,Martin S. Feather,Michael Fisher,Javier Ibanez-Guzman,Brian Logan,Joanna I. Olszewska,Signe Redfield
2024-11-21
Abstract:Autonomous systems use independent decision-making with only limited human intervention to accomplish goals in complex and unpredictable environments. As the autonomy technologies that underpin them continue to advance, these systems will find their way into an increasing number of applications in an ever wider range of settings. If we are to deploy them to perform safety-critical or mission-critical roles, it is imperative that we have justified confidence in their safe and correct operation. Verification is the process by which such confidence is established. However, autonomous systems pose challenges to existing verification practices. This paper highlights viewpoints of the Roadmap Working Group of the IEEE Robotics and Automation Society Technical Committee for Verification of Autonomous Systems, identifying these grand challenges, and providing a vision for future research efforts that will be needed to address them.
Robotics
What problem does this paper attempt to address?
### What problems does this paper attempt to solve? This paper aims to solve the major challenges faced in the verification process of autonomous systems (Autonomous Systems). With the continuous development of autonomous technology, these systems will be deployed in more and more application scenarios, especially in complex and unpredictable environments to perform safety - critical or mission - critical roles. To ensure that these systems can operate safely and correctly, confidence in their behavior must be established. Verification is the process of establishing this confidence, but the characteristics of autonomous systems pose new problems for existing verification methods. #### Specific problems include: 1. **Uncertainty**: - In particular, epistemic uncertainty due to limited knowledge makes it difficult to foresee all possible situations during the design and verification stages. For example, when the environment in which the system is deployed is different from the environment considered during design, or when the environment changes over time, many "unknown unknowns" will occur. - Modeling these unknown factors and defining appropriate behavior in such situations is a key challenge. 2. **Context**: - Many autonomous systems need to interact with humans or other agents and bear legal, ethical, and social responsibilities. These non - functional requirements have traditionally been regarded as non - functional requirements at the design stage, but as the autonomy of the system increases, these requirements become more important and need to be transformed into verifiable functional requirements. - Methods need to be developed to describe and reason about the physical or computational environment and the broader context (such as ethical, legal, and social factors) to ensure that the system makes correct decisions in various situations. 3. **Requirements**: - Formulating precise, consistent, and complete requirement specifications is a prerequisite for verification. For autonomous systems, it is very difficult to transform the actions that humans consider reasonable in specific situations into system requirements. - Development processes and standards are required to standardize requirements covering ethical, legal, and social expectations and convert them into specifications that can be used for verification. 4. **Semantics**: - Ensuring that the functional requirements of the system can guarantee the overall non - functional requirements usually involves different levels of detail and concerns. For example, from sensing pedestrians at the sensor level to deciding to stop and then performing actions, each component has different semantics. - Verifying across these semantic boundaries is a non - trivial task and requires more research. 5. **Operating Envelopes**: - When autonomous systems operate independently in complex environments, they may encounter unforeseen factors. Methods need to be developed to detect whether the system model still reflects the object or process it represents and to distinguish between cases where the system is outside its operating range and cases where the model representation is inaccurate. - Introducing the concept of Operational Design Domain (ODD) similar to that in the automotive field can help identify the conditions under which the system should operate normally and the situations where backup behaviors need to be adopted. 6. **Novel Verification Techniques**: - The programming methods of autonomous systems are very different from those of automated systems, including diagnostic subsystems, planners, schedulers, and reasoning engines. These differences require new verification techniques. - Decision - making is a core part of autonomous systems, but verifying complex decision - making processes remains an unsolved problem, especially in the case of combining multiple uncertainties. 7. **A Wider View of Verification**: - The presentation and communication of the verification process and its results cannot be one - size - fits - all, but need to be customized according to the recipients of the information. For example, users may only need to understand the behavior of the system, while developers need more detailed information. In summary, this paper proposes directions for future research to address these challenges in the verification of autonomous systems and ensure that these systems are not only capable but also safe and reliable.