STV+AGR: Towards Practical Verification of Strategic Ability Using Assume-Guarantee Reasoning

Damian Kurpiewski,Łukasz Mikulski,Wojciech Jamroga

2023-10-28

Abstract:We present a substantially expanded version of our tool STV for strategy synthesis and verification of strategic abilities. The new version provides a web interface and support for assume-guarantee verification of multi-agent systems.

Multiagent Systems

What problem does this paper attempt to address?

STV+Reductions: Towards Practical Verification of Strategic Ability Using Model Reductions

Damian Kurpiewski,Witold Pazderski,Wojciech Jamroga,Yan Kim

DOI: https://doi.org/10.5555/3463952.3464232

2023-10-28

Abstract:We present a substantially expanded version of our tool STV for strategy synthesis and verification of strategic abilities. The new version adds user-definable models and support for model reduction through partial order reduction and checking for bisimulation.

Logic in Computer Science,Multiagent Systems
Multi-Valued Verification of Strategic Ability

Wojciech Jamroga,Beata Konikowska,Damian Kurpiewski,Wojciech Penczek

DOI: https://doi.org/10.3233/FI-2020-1955

2023-10-31

Abstract:Some multi-agent scenarios call for the possibility of evaluating specifications in a richer domain of truth values. Examples include runtime monitoring of a temporal property over a growing prefix of an infinite path, inconsistency analysis in distributed databases, and verification methods that use incomplete anytime algorithms, such as bounded model checking. In this paper, we present multi-valued alternating-time temporal logic (mv-ATL*), an expressive logic to specify strategic abilities in multi-agent systems. It is well known that, for branching-time logics, a general method for model-independent translation from multi-valued to two-valued model checking exists. We show that the method cannot be directly extended to mv-ATL*. We also propose two ways of overcoming the problem. Firstly, we identify constraints on formulas for which the model-independent translation can be suitably adapted. Secondly, we present a model-dependent reduction that can be applied to all formulas of mv-ATL*. We show that, in all cases, the complexity of verification increases only linearly when new truth values are added to the evaluation domain. We also consider several examples that show possible applications of mv-ATL* and motivate its use for model checking multi-agent systems.

Multiagent Systems
Towards Modelling and Verification of Social Explainable AI

Damian Kurpiewski,Wojciech Jamroga,Teofil Sidoruk

2023-10-20

Abstract:Social Explainable AI (SAI) is a new direction in artificial intelligence that emphasises decentralisation, transparency, social context, and focus on the human users. SAI research is still at an early stage. Consequently, it concentrates on delivering the intended functionalities, but largely ignores the possibility of unwelcome behaviours due to malicious or erroneous activity. We propose that, in order to capture the breadth of relevant aspects, one can use models and logics of strategic ability, that have been developed in multi-agent systems. Using the STV model checker, we take the first step towards the formal modelling and verification of SAI environments, in particular of their resistance to various types of attacks by compromised AI modules.

Multiagent Systems
Scalable Verification of Strategy Logic through Three-valued Abstraction

Francesco Belardinelli,Angelo Ferrando,Wojciech Jamroga,Vadim Malvone,Aniello Murano

DOI: https://doi.org/10.48550/arXiv.2310.17219

2023-10-26

Multiagent Systems

Abstract:The model checking problem for multi-agent systems against Strategy Logic specifications is known to be non-elementary. On this logic several fragments have been defined to tackle this issue but at the expense of expressiveness. In this paper, we propose a three-valued semantics for Strategy Logic upon which we define an abstraction method. We show that the latter semantics is an approximation of the classic two-valued one for Strategy Logic. Furthermore, we extend MCMAS, an open-source model checker for multi-agent specifications, to incorporate our abstraction method and present some promising experimental results.
Strategic Abilities of Asynchronous Agents: Semantic Side Effects and How to Tame Them

Wojciech Jamroga,Wojciech Penczek,Teofil Sidoruk

2023-10-18

Abstract:Recently, we have proposed a framework for verification of agents' abilities in asynchronous multi-agent systems, together with an algorithm for automated reduction of models. The semantics was built on the modeling tradition of distributed systems. As we show here, this can sometimes lead to counterintuitive interpretation of formulas when reasoning about the outcome of strategies. First, the semantics disregards finite paths, and thus yields unnatural evaluation of strategies with deadlocks. Secondly, the semantic representations do not allow to capture the asymmetry between proactive agents and the recipients of their choices. We propose how to avoid the problems by a suitable extension of the representations and change of the execution semantics for asynchronous MAS. We also prove that the model reduction scheme still works in the modified framework.

Logic in Computer Science,Multiagent Systems
Automatic Verification of Competitive Stochastic Systems

Taolue Chen,Vojtěch Forejt,Marta Kwiatkowska,David Parker,Aistis Simaitis

DOI: https://doi.org/10.1007/s10703-013-0183-7

2013-01-01

Formal Methods in System Design

Abstract:We present automatic verification techniques for the modelling and analysis of probabilistic systems that incorporate competitive behaviour. These systems are modelled as turn-based stochastic multi-player games, in which the players can either collaborate or compete in order to achieve a particular goal. We define a temporal logic called rPATL for expressing quantitative properties of stochastic multi-player games. This logic allows us to reason about the collective ability of a set of players to achieve a goal relating to the probability of an event’s occurrence or the expected amount of cost/reward accumulated. We give an algorithm for verifying properties expressed in this logic and implement the techniques in a probabilistic model checker, as an extension of the PRISM tool. We demonstrate the applicability and efficiency of our methods by deploying them to analyse and detect potential weaknesses in a variety of large case studies, including algorithms for energy management in Microgrids and collective decision making for autonomous systems.
A Model Checker for Natural Strategic Ability

Marco Aruta,Vadim Malvone,Aniello Murano

2024-10-18

Abstract:In the last two decades, Alternating-time Temporal Logic (ATL) has been proved to be very useful in modeling strategic reasoning for Multi-Agent Systems (MAS). However, this logic struggles to capture the bounded rationality inherent in human decision-making processes. To overcome these limitations, Natural Alternating-time Temporal Logic (NatATL) has been recently introduced. As an extension of ATL, NatATL incorporates bounded memory constraints into agents' strategies, which allows to resemble human cognitive limitations. In this paper, we present a model checker tool for NatATL specifications - both for memoryless strategies and strategies with recall - integrated into VITAMIN, an open-source model checker designed specifically for MAS verification. By embedding NatATL into VITAMIN, we transform theoretical advancements into a practical verification framework, enabling comprehensive analysis and validation of strategic reasoning in complex multi-agent environments. Our novel tool paves the way for applications in areas such as explainable AI and human-in-the-loop systems, highlighting NatATL's substantial potential.

Multiagent Systems,Logic in Computer Science
QaSten: Integrating Quantitative Verification with Safety Analysis for AADL Model

Xiaomin Wei,Yunwei Dong,Hong Ye

DOI: https://doi.org/10.1109/TASE.2015.10

2015-01-01

Abstract:Quantitative verification is an effective technique for analyzing quantitative aspects of a safety critical system's design, and safety analysis is a significant aspect of safety critical system. However, they are often conducted separately. In this paper, we propose a new methodology, QaSten, fastens quantitative verification to safety analysis for Architecture Analysis and Design Language (AADL) model (including error model). QaSten formalizes a set of rigorous transformation rules that transform AADL model to PRISM model using formal method. In addition, QaSten can generate two safety property formulas automatically to check against the PRISM model for each hazardous state. Therefore, the occurrence probability of hazardous states can be calculated, which can help system designers understand the impact of parameters in the model. Furthermore, combining the probability and the severity of potential consequence of a hazardous state, QaSten determines the hazard risk acceptance level that can help engineers to identify critical hazard and modify or redesign architecture model to control it in an acceptable level. Two case studies, based on the Gas Leakage Alarm systems, are utilized to demonstrate QaSten's feasibility and effectiveness.
PRISM-games: a model checker for stochastic multi-player games

Taolue Chen,Vojtěch Forejt,Marta Kwiatkowska,David Parker,Aistis Simaitis

DOI: https://doi.org/10.1007/978-3-642-36742-7_13

2013-01-01

Abstract:We present PRISM-games, a model checker for stochastic multi-player games, which supports modelling, automated verification and strategy synthesis for probabilistic systems with competitive or cooperative behaviour. Models are described in a probabilistic extension of the Reactive Modules language and properties are expressed using rPATL, which extends the well-known logic ATL with operators to reason about probabilities, various reward-based measures, quantitative properties and precise bounds. The tool is based on the probabilistic model checker PRISM, benefiting from its existing user interface and simulator, whilst adding novel model checking algorithms for stochastic games, as well as functionality to synthesise optimal player strategies, explore or export them, and verify other properties under the specified strategy.
Expectation vs. Reality: Towards Verification of Psychological Games

Marta Kwiatkowska,Gethin Norman,David Parker,Gabriel Santos

2024-11-08

Abstract:Game theory provides an effective way to model strategic interactions among rational agents. In the context of formal verification, these ideas can be used to produce guarantees on the correctness of multi-agent systems, with a diverse range of applications from computer security to autonomous driving. Psychological games (PGs) were developed as a way to model and analyse agents with belief-dependent motivations, opening up the possibility to model how human emotions can influence behaviour. In PGs, players' utilities depend not only on what actually happens (which strategies players choose to adopt), but also on what the players had expected to happen (their belief as to the strategies that would be played). Despite receiving much attention in fields such as economics and psychology, very little consideration has been given to their applicability to problems in computer science, nor to practical algorithms and tool support. In this paper, we start to bridge that gap, proposing methods to solve PGs and implementing them within PRISM-games, a formal verification tool for stochastic games. We discuss how to model these games, highlight specific challenges for their analysis and illustrate the usefulness of our approach on several case studies, including human behaviour in traffic scenarios.

Computer Science and Game Theory,Artificial Intelligence,Multiagent Systems
SmartVerif: Push the Limit of Automation Capability of Verifying Security Protocols by Dynamic Strategies.

Yan Xiong,Cheng Su,Wenchao Huang,Fuyou Miao,Wansen Wang,Hengyi Ouyang

DOI: https://doi.org/10.1109/tdsc.2024.3368131

2020-01-01

Abstract:Current formal approaches have been successfully used to find design flaws in many security protocols. However, it is still challenging to automatically analyze protocols due to their large or infinite state spaces. In this paper, we propose SmartVerif, a novel and general framework that pushes the limit of automation capability of state-of-the-art verification approaches. The primary technical contribution is the dynamic strategy inside SmartVerif, which can be used to smartly search proof paths. Different from the non-trivial and error-prone design of existing static strategies, the design of our dynamic strategy is simple and flexible: it can automatically optimize itself according to the security protocols without any human intervention. With the optimized strategy, SmartVerif can localize and prove supporting lemmata, which leads to higher probability of success in verification. The insight of designing the strategy is that the node representing a supporting lemma is on an incorrect proof path with lower probability, when a random strategy is given. Hence, we implement the strategy around the insight by introducing a reinforcement learning algorithm. We also propose several methods to deal with other technical problems in implementing SmartVerif. Experimental results show that SmartVerif can automatically verify all security protocols studied in this paper. The case studies also validate the efficiency of our dynamic strategy.
Practical Model Reductions for Verification of Multi-Agent Systems

Wojciech Jamroga,Yan Kim

2023-10-18

Abstract:Formal verification of intelligent agents is often computationally infeasible due to state-space explosion. We present a tool for reducing the impact of the explosion by means of state abstraction that is (a) easy to use and understand by non-experts, and (b) agent-based in the sense that it operates on a modular representation of the system, rather than on its huge explicit state model.

Multiagent Systems
A stochastically verifiable autonomous control architecture with reasoning

Paolo Izzo,Hongyang Qu,Sandor M. Veres

DOI: https://doi.org/10.48550/arXiv.1611.03372

2016-11-11

Abstract:A new agent architecture called Limited Instruction Set Agent (LISA) is introduced for autonomous control. The new architecture is based on previous implementations of AgentSpeak and it is structurally simpler than its predecessors with the aim of facilitating design-time and run-time verification methods. The process of abstracting the LISA system to two different types of discrete probabilistic models (DTMC and MDP) is investigated and illustrated. The LISA system provides a tool for complete modelling of the agent and the environment for probabilistic verification. The agent program can be automatically compiled into a DTMC or a MDP model for verification with Prism. The automatically generated Prism model can be used for both design-time and run-time verification. The run-time verification is investigated and illustrated in the LISA system as an internal modelling mechanism for prediction of future outcomes.

Robotics,Artificial Intelligence,Software Engineering,Systems and Control
V-STaR: Training Verifiers for Self-Taught Reasoners

Arian Hosseini,Xingdi Yuan,Nikolay Malkin,Aaron Courville,Alessandro Sordoni,Rishabh Agarwal

2024-08-14

Abstract:Common self-improvement approaches for large language models (LLMs), such as STaR, iteratively fine-tune LLMs on self-generated solutions to improve their problem-solving ability. However, these approaches discard the large amounts of incorrect solutions generated during this process, potentially neglecting valuable information in such solutions. To address this shortcoming, we propose V-STaR that utilizes both the correct and incorrect solutions generated during the self-improvement process to train a verifier using DPO that judges correctness of model-generated solutions. This verifier is used at inference time to select one solution among many candidate solutions. Running V-STaR for multiple iterations results in progressively better reasoners and verifiers, delivering a 4% to 17% test accuracy improvement over existing self-improvement and verification approaches on common code generation and math reasoning benchmarks with LLaMA2 models.

Machine Learning,Artificial Intelligence,Computation and Language
VERIFAS: A Practical Verifier for Artifact Systems

Yuliang Li,Alin Deutsch,Victor Vianu

DOI: https://doi.org/10.48550/arXiv.1705.10007

2017-09-14

Abstract:Data-driven workflows, of which IBM's Business Artifacts are a prime exponent, have been successfully deployed in practice, adopted in industrial standards, and have spawned a rich body of research in academia, focused primarily on static analysis. The present research bridges the gap between the theory and practice of artifact verification with VERIFAS, the first implementation of practical significance of an artifact verifier with full support for unbounded data. VERIFAS verifies within seconds linear-time temporal properties over real-world and synthetic workflows of complexity in the range recommended by software engineering practice. Compared to our previous implementation based on the widely-used Spin model checker, VERIFAS not only supports a model with richer data manipulations but also outperforms it by over an order of magnitude. VERIFAS' good performance is due to a novel symbolic representation approach and a family of specialized optimizations.

Databases,Logic in Computer Science
Trusta: Reasoning about Assurance Cases with Formal Methods and Large Language Models

Zezhong Chen,Yuxin Deng,Wenjie Du

2023-09-22

Abstract:Assurance cases can be used to argue for the safety of products in safety engineering. In safety-critical areas, the construction of assurance cases is indispensable. Trustworthiness Derivation Trees (TDTs) enhance assurance cases by incorporating formal methods, rendering it possible for automatic reasoning about assurance cases. We present Trustworthiness Derivation Tree Analyzer (Trusta), a desktop application designed to automatically construct and verify TDTs. The tool has a built-in Prolog interpreter in its backend, and is supported by the constraint solvers Z3 and MONA. Therefore, it can solve constraints about logical formulas involving arithmetic, sets, Horn clauses etc. Trusta also utilizes large language models to make the creation and evaluation of assurance cases more convenient. It allows for interactive human examination and modification. We evaluated top language models like ChatGPT-3.5, ChatGPT-4, and PaLM 2 for generating assurance cases. Our tests showed a 50%-80% similarity between machine-generated and human-created cases. In addition, Trusta can extract formal constraints from text in natural languages, facilitating an easier interpretation and validation process. This extraction is subject to human review and correction, blending the best of automated efficiency with human insight. To our knowledge, this marks the first integration of large language models in automatic creating and reasoning about assurance cases, bringing a novel approach to a traditional challenge. Through several industrial case studies, Trusta has proven to quickly find some subtle issues that are typically missed in manual inspection, demonstrating its practical value in enhancing the assurance case development process.

Software Engineering,Artificial Intelligence
Capacity ATL

Gabriel Ballot,Vadim Malvone,Jean Leneutre,Youssef Laarouchi

2023-08-22

Abstract:Model checking strategic abilities was successfully developed and applied since the early 2000s to ensure properties in Multi-Agent System. In this paper, we introduce the notion of capacities giving different abilities to an agent. This applies naturally to systems where multiple entities can play the same role in the game, such as different client versions in protocol analysis, different robots in heterogeneous fleets, different personality traits in social structure modeling, or different attacker profiles in a cybersecurity setting. With the capacity of other agents being unknown at the beginning of the game, the longstanding problems of imperfect information arise. Our contribution is the following: (i) we define a new class of concurrent game structures where the agents have different capacities that modify their action list and (ii) we introduce a logic extending Alternating-time Temporal Logic to reason about these games.

Multiagent Systems
A SAT-based approach to rigorous verification of Bayesian networks

Ignacy Stępka,Nicholas Gisolfi,Artur Dubrawski

2024-08-02

Abstract:Recent advancements in machine learning have accelerated its widespread adoption across various real-world applications. However, in safety-critical domains, the deployment of machine learning models is riddled with challenges due to their complexity, lack of interpretability, and absence of formal guarantees regarding their behavior. In this paper, we introduce a verification framework tailored for Bayesian networks, designed to address these drawbacks. Our framework comprises two key components: (1) a two-step compilation and encoding scheme that translates Bayesian networks into Boolean logic literals, and (2) formal verification queries that leverage these literals to verify various properties encoded as constraints. Specifically, we introduce two verification queries: if-then rules (ITR) and feature monotonicity (FMO). We benchmark the efficiency of our verification scheme and demonstrate its practical utility in real-world scenarios.

Artificial Intelligence,Logic in Computer Science
RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement

Jinhao Jiang,Jiayi Chen,Junyi Li,Ruiyang Ren,Shijie Wang,Wayne Xin Zhao,Yang Song,Tao Zhang

2024-12-17

Abstract:Existing large language models (LLMs) show exceptional problem-solving capabilities but might struggle with complex reasoning tasks. Despite the successes of chain-of-thought and tree-based search methods, they mainly depend on the internal knowledge of LLMs to search over intermediate reasoning steps, limited to dealing with simple tasks involving fewer reasoning steps. In this paper, we propose \textbf{RAG-Star}, a novel RAG approach that integrates the retrieved information to guide the tree-based deliberative reasoning process that relies on the inherent knowledge of LLMs. By leveraging Monte Carlo Tree Search, RAG-Star iteratively plans intermediate sub-queries and answers for reasoning based on the LLM itself. To consolidate internal and external knowledge, we propose an retrieval-augmented verification that utilizes query- and answer-aware reward modeling to provide feedback for the inherent reasoning of LLMs. Our experiments involving Llama-3.1-8B-Instruct and GPT-4o demonstrate that RAG-Star significantly outperforms previous RAG and reasoning methods.

Computation and Language,Artificial Intelligence
Modular Formalisation and Verification of STV Algorithms

Milad K. Ghale,Rajeev Goré,Dirk Pattinson,Mukesh Tiwari

DOI: https://doi.org/10.1007/978-3-030-00419-4_4

2018-01-01

Abstract:We introduce a formal, modular framework that captures a large number of different instances of the Single Transferable Vote (STV) counting scheme in a uniform way. The framework requires that each instance defines the precise mechanism of counting and transferring ballots, electing and eliminating candidates. From formal proofs of basic sanity conditions for each mechanism inside the Coq theorem prover, we then synthesise code that implements the given scheme in a provably correct way and produces a universally verifiable certificate of the count. We have applied this to various variations of STV, including several used in Australian parliamentary elections and demonstrated the feasibility of our approach by means of real-world case studies.

STV+AGR: Towards Practical Verification of Strategic Ability Using Assume-Guarantee Reasoning

STV+Reductions: Towards Practical Verification of Strategic Ability Using Model Reductions

Multi-Valued Verification of Strategic Ability

Towards Modelling and Verification of Social Explainable AI

Scalable Verification of Strategy Logic through Three-valued Abstraction

Strategic Abilities of Asynchronous Agents: Semantic Side Effects and How to Tame Them

Automatic Verification of Competitive Stochastic Systems

A Model Checker for Natural Strategic Ability

QaSten: Integrating Quantitative Verification with Safety Analysis for AADL Model

PRISM-games: a model checker for stochastic multi-player games

Expectation vs. Reality: Towards Verification of Psychological Games

SmartVerif: Push the Limit of Automation Capability of Verifying Security Protocols by Dynamic Strategies.

Practical Model Reductions for Verification of Multi-Agent Systems

A stochastically verifiable autonomous control architecture with reasoning

V-STaR: Training Verifiers for Self-Taught Reasoners

VERIFAS: A Practical Verifier for Artifact Systems

Trusta: Reasoning about Assurance Cases with Formal Methods and Large Language Models

Capacity ATL

A SAT-based approach to rigorous verification of Bayesian networks

RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement

Modular Formalisation and Verification of STV Algorithms