Abstract:Cao's work shows that, by defining an α-dependent equivalent infinitesimal generator Aα, a semi-Markov decision process (SMDP) with both average- and discounted-cost criteria can be treated as an α-equivalent Markov decision process (MDP), and the performance potential theory can also be developed for SMDPs. In this work, we focus on establishing error bounds for potential and Aα-based iterative optimization methods. First, we introduce an α-uniformized Markov chain (UMC) for a SMDP via Aα and a uniformized parameter, and show their relations. Especially, we obtain that their performance potentials, as solutions of corresponding Poisson equations, are proportional, so that the studies of a SMDP and the α-UMC based on potentials are unified. Using these relations, we derive the error bounds for a potential-based policy-iteration algorithm and a value-iteration algorithm, respectively, when there exist various calculation errors. The obtained results can be applied directly to the special models, i.e., continuous-time MDPs and Markov chains, and can be extended to some simulation-based optimization methods such as reinforcement learning and neurodynamic programming, where estimation errors or approximation errors are common cases. Finally, we give an application example on the look-ahead control of a conveyor-serviced production station (CSPS), and show the corresponding error bounds.

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Optimal Stationary Policies for Semi-Markov Control Processes with Discounted-Cost Criteria

Error bounds of optimization algorithms for semi-Markov decision processes

Relations Between Discounted Models and Average Models for Semi-Markov Decision Processes

Simulation Optimization Algorithm for SMDPs with Parameterized Randomized Stationary Policies

Mixed Markov Decision Processes in a Semi-Markov Environment with Discounted Criterion

Optimization Algorithms for Semi-Markov Control Processes with Average Criteria

Sensitivity Analysis and Performance Optimization of Semi-Markov Processes Based on Performance Potentials

Performance Optimization of Continuous-Time Markov Control Processes Based on Performance Potentials

Optimal Stationary Policies for a Class of Countable Markov Control Processes

A Simulation Optimization Algorithm for CTMDPs Based on Randomized Stationary Policies

Parallel Optimization for Markov Control Processes Based on Performance Potentials Simulation

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

Potential Based Optimization Algorithm Of Constrained Markov Decision Processes

OPTIMALITY STRATEGY OF AVERAGE COST BASED PERFORMANCE POTENTIALS FOR MARKOV CONTROL PROCESS

Towards Analysis Of Semi-Markov Decision Processes

Analysis for Some Properties of Discrete Time Markov Decision Processes

The Estimation of the Semi-Markov Processes Performance Potentials Based on Parallel Simulation

The Finiteness of the Reward Function and the Optimal Value Function in Markov Decision Processes

Partially Observable Markov Decision Processes and Performance Sensitivity Analysis