Abstract:Cao's work shows that, by defining an α-dependent equivalent infinitesimal generator Aα, a semi-Markov decision process (SMDP) with both average- and discounted-cost criteria can be treated as an α-equivalent Markov decision process (MDP), and the performance potential theory can also be developed for SMDPs. In this work, we focus on establishing error bounds for potential and Aα-based iterative optimization methods. First, we introduce an α-uniformized Markov chain (UMC) for a SMDP via Aα and a uniformized parameter, and show their relations. Especially, we obtain that their performance potentials, as solutions of corresponding Poisson equations, are proportional, so that the studies of a SMDP and the α-UMC based on potentials are unified. Using these relations, we derive the error bounds for a potential-based policy-iteration algorithm and a value-iteration algorithm, respectively, when there exist various calculation errors. The obtained results can be applied directly to the special models, i.e., continuous-time MDPs and Markov chains, and can be extended to some simulation-based optimization methods such as reinforcement learning and neurodynamic programming, where estimation errors or approximation errors are common cases. Finally, we give an application example on the look-ahead control of a conveyor-serviced production station (CSPS), and show the corresponding error bounds.

A Survey on Semi-Markov Decision Processes

A survey of recent results on continuous-time Markov decision processes

Towards Analysis Of Semi-Markov Decision Processes

Mixed Markov Decision Processes in a Semi-Markov Environment with Discounted Criterion

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

The Finiteness of the Reward Function and the Optimal Value Function in Markov Decision Processes

Error bounds of optimization algorithms for semi-Markov decision processes

Markov Decision Processes with Risk-Sensitive Criteria: An Overview

Markov Decision Processes under Risk Sensitivity: A Discount Vanishing Approach

Semi-Infinitely Constrained Markov Decision Processes and Efficient Reinforcement Learning

Relations Between Discounted Models and Average Models for Semi-Markov Decision Processes

Discrete Type Shock Semi-Markov Decision Processes with Borel State Space

Continuous Time Markov Decision Processes with Discounted Moment Criterion

Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

Optimal Stationary Policies for Semi-Markov Control Processes with Discounted-Cost Criteria

Semi-Infinitely Constrained Markov Decision Processes and Provably Efficient Reinforcement Learning.

Continuous Time Markov Decision Processes with Nonuniformly Bounded Transition Rate: Expected Total Rewards

Markov Decision Processes with Incomplete Information and Semi-Uniform Feller Transition Probabilities