Abstract:Cao's work shows that, by defining an α-dependent equivalent infinitesimal generator Aα, a semi-Markov decision process (SMDP) with both average- and discounted-cost criteria can be treated as an α-equivalent Markov decision process (MDP), and the performance potential theory can also be developed for SMDPs. In this work, we focus on establishing error bounds for potential and Aα-based iterative optimization methods. First, we introduce an α-uniformized Markov chain (UMC) for a SMDP via Aα and a uniformized parameter, and show their relations. Especially, we obtain that their performance potentials, as solutions of corresponding Poisson equations, are proportional, so that the studies of a SMDP and the α-UMC based on potentials are unified. Using these relations, we derive the error bounds for a potential-based policy-iteration algorithm and a value-iteration algorithm, respectively, when there exist various calculation errors. The obtained results can be applied directly to the special models, i.e., continuous-time MDPs and Markov chains, and can be extended to some simulation-based optimization methods such as reinforcement learning and neurodynamic programming, where estimation errors or approximation errors are common cases. Finally, we give an application example on the look-ahead control of a conveyor-serviced production station (CSPS), and show the corresponding error bounds.

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

Optimal Stationary Policies for Semi-Markov Control Processes with Discounted-Cost Criteria

Error bounds of optimization algorithms for semi-Markov decision processes

Markov Decision Problems with Unbounded Transition Rates under Discounted-Cost Performance Criteria

Mixed Markov Decision Processes in a Semi-Markov Environment with Discounted Criterion

A unified approach for semi-Markov decision processes with discounted and average reward criteria

Relations Between Discounted Models and Average Models for Semi-Markov Decision Processes

Optimal Stopping Time on Discounted Semi-Markov Processes

Performance Optimization Algorithms Based on Potentials for Semi-Markov Control Processes

Simulation Optimization Algorithm for SMDPs with Parameterized Randomized Stationary Policies

Optimization Algorithms for Semi-Markov Control Processes with Average Criteria

A Survey on Semi-Markov Decision Processes

Markov Decision Processes with State-Dependent Discount Factors and Unbounded Rewards/costs.

Semi-Markov Decision Processes with Variance Minimization Criterion

A Unified Approach to Markov Decision Problems and Performance Sensitivity Analysis with Discounted and Average Criteria: Multichain Cases

Sensitivity Analysis and Performance Optimization of Semi-Markov Processes Based on Performance Potentials

First Passage Models for Denumerable Semi-Markov Decision Processes with Nonnegative Discounted Costs

Finite Horizon Semi-Markov Decision Processes with Application to Maintenance Systems.

Performance Optimization of Continuous-Time Markov Control Processes Based on Performance Potentials

Risk-sensitive Semi-Markov Decision Processes with General Utilities and Multiple Criteria