Abstract:We consider the problem of computing the maximal probability of satisfying an ω -regular specification for stochastic, continuous-state, nonlinear systems evolving in discrete time. The problem reduces, after automata-theoretic constructions, to finding the maximal probability of satisfying a parity condition on a (possibly hybrid) state space. While characterizing the exact satisfaction probability is open, we show that a lower bound on this probability can be obtained by (I) computing an under-approximation of the qualitative winning region, i.e., states from which the parity condition can be enforced almost surely, and (II) computing the maximal probability of reaching this qualitative winning region. The heart of our approach is a technique to symbolically compute the under-approximation of the qualitative winning region in step (I) via a finite-state abstraction of the original system as a 2 1 2 -player parity game. Our abstraction procedure uses only the support of the probabilistic evolution; it does not use precise numerical transition probabilities. We prove that the winning set in the abstract 2 1 2 -player game induces an under-approximation of the qualitative winning region in the original synthesis problem, along with a policy to solve it. By combining these contributions with (a) a symbolic fixpoint algorithm to solve 2 1 2 -player games and (b) existing techniques for reachability policy synthesis in stochastic nonlinear systems, we get an abstraction-based algorithm for finding a lower bound on the maximal satisfaction probability. We have implemented the abstraction-based algorithm in Mascot-SDS, where we combined the outlined abstraction step with our tool Genie (Majumdar et al., 2023) that solves 2 1 2 -player parity games (through a reduction to Rabin games) more efficiently than existing algorithms. We evaluated our implementation on the nonlinear model of a perturbed bistable switch from the literature. We show empirically that the lower bound on the winning region computed by our approach is precise, by comparing against an over-approximation of the qualitative winning region. Moreover, our implementation outperforms a recently proposed tool for solving this problem by a large margin.

Tableaux for Policy Synthesis for MDPs with PCTL* Constraints

Strong Simple Policies for POMDPs

Policy Synthesis for Factored MDPs with Graph Temporal Logic Specifications

Controller synthesis for linear temporal logic and steady-state specifications

Policies Grow on Trees: Model Checking Families of MDPs

Bounded Policy Synthesis for POMDPs with Safe-Reachability Objectives

Learning Robust Policies for Uncertain Parametric Markov Decision Processes

Statistically Model Checking PCTL Specifications on Markov Decision Processes via Reinforcement Learning

Strategy Synthesis in POMDPs via Game-Based Abstractions

Probabilistic Plan Synthesis for Coupled Multi-Agent Systems

Search and Explore: Symbiotic Policy Synthesis in POMDPs

Deductive Controller Synthesis for Probabilistic Hyperproperties

1-2-3-Go! Policy Synthesis for Parameterized Markov Decision Processes via Decision-Tree Learning and Generalization

Symbolic control for stochastic systems via finite parity games

Negotiating the Probabilistic Satisfaction of Temporal Logic Motion Specifications

Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guarantees

Verifiable RNN-Based Policies for POMDPs Under Temporal Logic Constraints

Controlled Markov Processes With Safety State Constraints

Safety-Constrained Reinforcement Learning for MDPs

Entropy Rate Maximization of Markov Decision Processes under Linear Temporal Logic Tasks

Stochastic Finite State Control of POMDPs with LTL Specifications