Abstract:This paper considers a Markov decision process (MDP) model with safety state constraints, which specify polytopic invariance constraints on the state probability distribution (pd) for all time epochs. Typically, in the MDP framework, safety is addressed indirectly by penalizing failure states through the reward function. However, such an approach does not allow imposing hard constraints on the state pd, which could be an issue for practical applications where the chance of failure must be limited to prescribed bounds. In this paper, we explicitly separate state constraints from the reward function. We provide analysis and synthesis methods to impose generalized safety constraints at all time epochs, unlike current constrained MDP approaches where such constraints can only be imposed on the stationary distributions. We show that, contrary to the unconstrained MDP policies, optimal safe MDP policies depend on the initial state pd. We present novel algorithms for both finite- and infinite-horizon MDPs to synthesize feasible decision-making policies that satisfy safety constraints for all time epochs and ensure that the performance is above a computable lower bound. Linear programming implementations of the proposed algorithms are developed, which are formulated by using the duality theory of convex optimization. A swarm control simulation example is also provided to demonstrate the use of proposed algorithms.

Towards Analysis Of Semi-Markov Decision Processes

A Survey on Semi-Markov Decision Processes

Constrained Semi-Markov Decision Processes with Ratio and Time Expected Average Criteria in Polish Spaces

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Mixed Markov Decision Processes in a Semi-Markov Environment with Discounted Criterion

Discrete Type Shock Semi-Markov Decision Processes with Borel State Space

A unified approach for semi-Markov decision processes with discounted and average reward criteria

Error bounds of optimization algorithms for semi-Markov decision processes

Optimal Stopping Time on Discounted Semi-Markov Processes

Risk-sensitive Semi-Markov Decision Processes with General Utilities and Multiple Criteria

Optimal Time-Abstract Schedulers for CTMDPs and Markov Games

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

Finite Horizon Semi-Markov Decision Processes with Application to Maintenance Systems.

A Unified Approach to Time-Aggregated Markov Decision Processes.

Solving Markov Decision Processes with Reachability Characterization from Mean First Passage Times

Finite Horizon Semi-Markov Decision Processes with Multiple Constraints

Optimal Stopping Time on Semi-Markov Processes with Finite Horizon

Reachability and Safety Objectives in Markov Decision Processes on Long but Finite Horizons

Analytical Solution to A Discrete-Time Model for Dynamic Learning and Decision-Making

A Semimartingale Characterization of Average Optimal Stationary Policies for Markov Decision Processes

Controlled Markov Processes With Safety State Constraints