Abstract:Synthesis from linear temporal logic (LTL) specifications provides assured controllers for systems operating in stochastic and potentially adversarial environments. Automatic synthesis tools, however, require a model of the environment to construct controllers. In this work, we introduce a model-free reinforcement learning (RL) approach to derive controllers from given LTL specifications even when the environment is completely unknown. We model the problem as a stochastic game (SG) between the controller and the adversarial environment; we then learn optimal control strategies that maximize the probability of satisfying the LTL specifications against the worst-case environment behavior. We first construct a product game using the deterministic parity automaton (DPA) translated from the given LTL specification. By deriving distinct rewards and discount factors from the acceptance condition of the DPA, we reduce the maximization of the worst-case probability of satisfying the LTL specification into the maximization of a discounted reward objective in the product game; this enables the use of model-free RL algorithms to learn an optimal controller strategy. To deal with the common scalability problems when the number of sets defining the acceptance condition of the DPA (usually referred as colors), is large, we propose a lazy color generation method where distinct rewards and discount factors are utilized only when needed, and an approximate method where the controller eventually focuses on only one color. In several case studies, we show that our approach is scalable to a wide range of LTL formulas, significantly outperforming existing methods for learning controllers from LTL specifications in SGs.

Learning Temporal Properties is NP-hard

The Complexity of Learning Temporal Properties

Learning Branching-Time Properties in CTL and ATL via Constraint Solving

Learning Temporal Logic Properties: an Overview of Two Recent Methods

HyperLTL Satisfiability Is Highly Undecidable, HyperCTL* is Even Harder

Inferring Properties in Computation Tree Logic

Constrained LTL Specification Learning from Examples

Hardness of Learning Boolean Functions from Label Proportions

DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications

A logic for temporal conditionals and a solution to the Sea Battle Puzzle

Learning Probabilistic Temporal Safety Properties from Examples in Relational Domains

Learning Hidden Subgoals under Temporal Ordering Constraints in Reinforcement Learning

Properly Learning Decision Trees with Queries Is NP-Hard

Reinforcement Learning with Temporal Logic Constraints for Partially-Observable Markov Decision Processes

Pac-learning Recursive Logic Programs: Negative Results

Counting CTL

Semantics for Linear-time Temporal Logic with Finite Observations

LTL learning on GPUs

Learning Optimal Strategies for Temporal Tasks in Stochastic Games

Deep Reinforcement Learning with Temporal Logics

On the decidability and complexity of Metric Temporal Logic over finite words