Abstract:This paper considers a Markov decision process (MDP) model with safety state constraints, which specify polytopic invariance constraints on the state probability distribution (pd) for all time epochs. Typically, in the MDP framework, safety is addressed indirectly by penalizing failure states through the reward function. However, such an approach does not allow imposing hard constraints on the state pd, which could be an issue for practical applications where the chance of failure must be limited to prescribed bounds. In this paper, we explicitly separate state constraints from the reward function. We provide analysis and synthesis methods to impose generalized safety constraints at all time epochs, unlike current constrained MDP approaches where such constraints can only be imposed on the stationary distributions. We show that, contrary to the unconstrained MDP policies, optimal safe MDP policies depend on the initial state pd. We present novel algorithms for both finite- and infinite-horizon MDPs to synthesize feasible decision-making policies that satisfy safety constraints for all time epochs and ensure that the performance is above a computable lower bound. Linear programming implementations of the proposed algorithms are developed, which are formulated by using the duality theory of convex optimization. A swarm control simulation example is also provided to demonstrate the use of proposed algorithms.

Controlled Semi-Markov Chains with Risk-Sensitive Average Cost Criterion

Characterization of the optimal average cost in Markov decision chains driven by a risk-seeking controller

Risk-Sensitive Optimal Control in Communicating Average Markov Decision Chains

Solution to the risk-sensitive average cost optimality equation in a class of Markov decision processes with finite state space

On Average Risk-sensitive Markov Control Processes

Risk-sensitive control of continuous time Markov chains

Controlled Markov Chains with AVaR Criteria for Unbounded Costs

Risk-sensitive Markov control processes

Risk-Sensitive Average Markov Decision Processes in General Spaces

Optimization Algorithms for Semi-Markov Control Processes with Average Criteria

Risk-Sensitive Stopping Problems for Continuous-Time Markov Chains

Optimal Stationary Policies for Semi-Markov Control Processes with Discounted-Cost Criteria

Local Poisson Equations Associated with Discrete-Time Markov Control Processes

Risk-Averse Control of Undiscounted Transient Markov Models

Markov decision processes with risk-sensitive criteria: an overview

Networked guaranteed cost control for uncertain discrete-time markovian jump linear systems

Risk-Sensitivity Vanishing Limit for Controlled Markov Processes

A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity

Algorithms for optimization and stabilization of controlled Markov chains

Controlled Markov Processes With Safety State Constraints