Abstract:This paper concerns studies on continuous-time controlled Markov chains, that is, continuous-time Markov decision processes with a denumerable state space, with respect to the discounted cost criterion. The cost and transition rates are allowed to be unbounded and the action set is a Borel space. We first study control problems in the class of deterministic stationary policies and give very weak conditions under which the existence of epsilon-optimal (epsilon greater than or equal to 0) policies is proved using the construction of a minimum Q-process. Then we further consider control problems in the class of randomized Markov policies for (1) regular and (2) nonregular Q-processes. To study case (1), first we present a new necessary and sufficient condition for a nonhomogeneous Q-process to be regular. This regularity condition, together with the extended generator of a nonhomogeneous Markov process, is used to prove the existence of epsilon-optimal stationary policies. Our results for case (1) are illustrated by a Schlogl model with a controlled diffusion. For case (2), we obtain a similar result using Kolmogorov's forward equation for the minimum Q-process and we also present an example in which our assumptions are satisfied, but those used in the previous literature fail to hold.

Denumerable Continuous-Time Markov Decision Processes with Multiconstraints on Average Costs

Constrained Continuous-Time Markov Decision Processes with Average Criteria

Discounted Continuous-Time Markov Decision Processes with Constraints: Unbounded Transition and Loss Rates

Constrained Optimization for Average Cost Continuous-Time Markov Decision Processes.

Constrained Continuous-Time Markov Control Processes with Discounted Criteria

Constrained Total Undiscounted Continuous-Time Markov Decision Processes

A Note on Optimality Conditions for Continuous-Time Markov Decision Processes with Average Cost Criterion

Continuous-Time Controlled Markov Chains

New Discount and Average Optimality Conditions for Continuous-Time Markov Decision Processes

Unbounded Cost Markov Decision Processes with Limsup and Liminf Average Criteria: New Conditions

Multiconstrained Finite-Horizon Piecewise Deterministic Markov Decision Processes with Unbounded Transition Rates

Risk-sensitive Average Continuous-Time Markov Decision Processes with Unbounded Transition and Cost Rates.

Total Reward Criteria for Unconstrained/constrained Continuous-Time Markov Decision Processes.

Convergence of Markov Decision Processes with Constraints and State-Action Dependent Discount Factors

Absorbing Continuous-Time Markov Decision Processes with Total Cost Criteria

Optimality of Mixed Policies for Average Continuous-Time Markov Decision Processes with Constraints

A New Condition for the Existence of Optimal Stationary Policies in Denumerable State Average Cost Continuous Time Markov Decision Processes with Unbounded Cost and Transition Rates

Denumerable-state Continuous-Time Markov Decision Processes with Unbounded Transition and Reward Rates under the Discounted Criterion

On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

Average cost Markov decision processes with countable state spaces

Discounted Continuous-Time Constrained Markov Decision Processes in Polish Spaces