Abstract:Cao's work shows that, by defining an α-dependent equivalent infinitesimal generator Aα, a semi-Markov decision process (SMDP) with both average- and discounted-cost criteria can be treated as an α-equivalent Markov decision process (MDP), and the performance potential theory can also be developed for SMDPs. In this work, we focus on establishing error bounds for potential and Aα-based iterative optimization methods. First, we introduce an α-uniformized Markov chain (UMC) for a SMDP via Aα and a uniformized parameter, and show their relations. Especially, we obtain that their performance potentials, as solutions of corresponding Poisson equations, are proportional, so that the studies of a SMDP and the α-UMC based on potentials are unified. Using these relations, we derive the error bounds for a potential-based policy-iteration algorithm and a value-iteration algorithm, respectively, when there exist various calculation errors. The obtained results can be applied directly to the special models, i.e., continuous-time MDPs and Markov chains, and can be extended to some simulation-based optimization methods such as reinforcement learning and neurodynamic programming, where estimation errors or approximation errors are common cases. Finally, we give an application example on the look-ahead control of a conveyor-serviced production station (CSPS), and show the corresponding error bounds.

Relations Between Discounted Models and Average Models for Semi-Markov Decision Processes

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

Mixed Markov Decision Processes in a Semi-Markov Environment with Discounted Criterion

Optimal Stationary Policies for Semi-Markov Control Processes with Discounted-Cost Criteria

Optimization Algorithms for Semi-Markov Control Processes with Average Criteria

Error bounds of optimization algorithms for semi-Markov decision processes

A Study of the Effect of the Implicit Performance Model on Managerial Decision

On Average Optimality for Non-Stationary Markov Decision Processes in Borel Spaces

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

Modeling Freight-Sharing Platform Operations for Optimal Compensation Strategy Using Markov Decision Processes

OPTIMALITY STRATEGY OF AVERAGE COST BASED PERFORMANCE POTENTIALS FOR MARKOV CONTROL PROCESS

Analysis for Some Properties of Discrete Time Markov Decision Processes

Sensitivity Analysis and Performance Optimization of Semi-Markov Processes Based on Performance Potentials

Towards Analysis Of Semi-Markov Decision Processes

Potential Based Optimization Algorithm Of Constrained Markov Decision Processes

Optimal Replacement of a System According Toa Semi-Markov Decision Process in a Semi-Markov Environment

Performance Optimization of Continuous-Time Markov Control Processes Based on Performance Potentials

Markov Decision Processes under Risk Sensitivity: A Discount Vanishing Approach

Average Optimality in Markov Decision Processes with Unbounded Rewards

On the optimality equation for average cost Markov decision processes and its validity for inventory control