Abstract:This paper is devoted to studying constrained continuous-time Markov decision processes (MDPs) in the class of randomized policies depending on state histories. The transition rates may be unbounded, the reward and costs are admitted to be unbounded from above and from below, and the state and action spaces are Polish spaces. The optimality criterion to be maximized is the expected discounted rewards, and the constraints can be imposed on the expected discounted costs. First, we give conditions for the nonexplosion of underlying processes and the finiteness of the expected discounted rewards/costs. Second, using a technique of occupation measures, we prove that the constrained optimality of continuous-time MDPs can be transformed to an equivalent (optimality) problem over a class of probability measures. Based on the equivalent problem and a so-called $\bar{w}$-weak convergence of probability measures developed in this paper, we show the existence of a constrained optimal policy. Third, by providing a linear programming formulation of the equivalent problem, we show the solvability of constrained optimal policies. Finally, we use two computable examples to illustrate our main results.

Multiconstrained Finite-Horizon Piecewise Deterministic Markov Decision Processes with Unbounded Transition Rates

Discounted Continuous-Time Markov Decision Processes with Constraints: Unbounded Transition and Loss Rates

Denumerable Continuous-Time Markov Decision Processes with Multiconstraints on Average Costs

Risk-sensitive Infinite-Horizon Discounted Piecewise Deterministic Markov Decision Processes

Risk probability optimization of finite horizon piecewise deterministic Markov decision processes

Finite Horizon Semi-Markov Decision Processes with Multiple Constraints

Constrained Total Undiscounted Continuous-Time Markov Decision Processes

Finite-horizon Optimality for Continuous-Time Markov Decision Processes with Unbounded Transition Rates

Risk-sensitive Average Continuous-Time Markov Decision Processes with Unbounded Transition and Cost Rates.

Discounted Continuous-Time Constrained Markov Decision Processes in Polish Spaces

Constrained Optimality for Finite Horizon Semi-Markov Decision Processes in Polish Spaces

Risk-Sensitive Discounted Continuous-Time Markov Decision Processes with Unbounded Rates.

Convergence of Markov Decision Processes with Constraints and State-Action Dependent Discount Factors

First Passage Risk Probability Minimization for Piecewise Deterministic Markov Decision Processes

Risk-sensitive Continuous-Time Markov Decision Processes with Unbounded Rates and Borel Spaces

Finite Horizon Risk-Sensitive Continuous-Time Markov Decision Processes with Unbounded Transition and Cost Rates

On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

Constrained Markov Decision Processes with First Passage Criteria

Constrained Markov Decision Processes with Non-constant Discount Factor

Constrained Continuous-Time Markov Decision Processes with Average Criteria

Risk-sensitive Semi-Markov Decision Processes with General Utilities and Multiple Criteria