Abstract:This paper deals with continuous-time Markov decision processes in Polish spaces, under the discounted and average cost criteria. All underlying Markov processes are determined by given transition rates which are allowed to be unbounded , and the costs are assumed to be bounded below . By introducing an occupation measure of a randomized Markov policy and analyzing properties of occupation measures, we first show that the family of all randomized stationary policies is ‘sufficient’ within the class of all randomized Markov policies. Then, under the semicontinuity and compactness conditions, we prove the existence of a discounted cost optimal stationary policy by providing a value iteration technique. Moreover, by developing a new average cost, minimum nonnegative solution method, we prove the existence of an average cost optimal stationary policy under some reasonably mild conditions. Finally, we use some examples to illustrate applications of our results. Except that the costs are assumed to be bounded below, the conditions for the existence of discounted cost (or average cost) optimal policies are much weaker than those in the previous literature, and the minimum nonnegative solution approach is new .

First Passage Optimality for Continuous-Time Markov Decision Processes with Varying Discount Factors and History-Dependent Policies

Finite-horizon Optimality for Continuous-Time Markov Decision Processes with Unbounded Transition Rates

On the First Passage G-Mean-variance Optimality for Discounted Continuous-Time Markov Decision Processes.

First Passage Models for Denumerable Semi-Markov Decision Processes with Nonnegative Discounted Costs

The Risk Probability Criterion for Discounted Continuous-Time Markov Decision Processes

Continuous-Time Markov Decision Processes with State-Dependent Discount Factors

First Passage Markov Decision Processes with Constraints and Varying Discount Factors

Continuous-Time Markov Decision Processes with Unbounded Transition and Discounted-Reward Rates

Risk-sensitive Continuous-Time Markov Decision Processes with Unbounded Rates and Borel Spaces

Denumerable-state Continuous-Time Markov Decision Processes with Unbounded Transition and Reward Rates under the Discounted Criterion

Constrained Markov Decision Processes with First Passage Criteria

Risk-Sensitive Discounted Continuous-Time Markov Decision Processes with Unbounded Rates.

Discounted Optimality for Continuous-Time Markov Decision Processes in Polish Spaces

Mean-variance optimality for semi-Markov decision processes under first passage criteria.

Optimal Risk Probability for First Passage Models in Semi-Markov Decision Processes

New Discount and Average Optimality Conditions for Continuous-Time Markov Decision Processes

Markov Decision Processes with State-Dependent Discount Factors and Unbounded Rewards/costs.

Continuous-Time Markov Decision Processes with Discounted Rewards: the Case of Polish Spaces

First Passage Problems for Nonstationary Discrete-Time Stochastic Control Systems

Discrete-time Zero-Sum Markov Games with First Passage Criteria

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards