Abstract:Cao's work shows that, by defining an α-dependent equivalent infinitesimal generator Aα, a semi-Markov decision process (SMDP) with both average- and discounted-cost criteria can be treated as an α-equivalent Markov decision process (MDP), and the performance potential theory can also be developed for SMDPs. In this work, we focus on establishing error bounds for potential and Aα-based iterative optimization methods. First, we introduce an α-uniformized Markov chain (UMC) for a SMDP via Aα and a uniformized parameter, and show their relations. Especially, we obtain that their performance potentials, as solutions of corresponding Poisson equations, are proportional, so that the studies of a SMDP and the α-UMC based on potentials are unified. Using these relations, we derive the error bounds for a potential-based policy-iteration algorithm and a value-iteration algorithm, respectively, when there exist various calculation errors. The obtained results can be applied directly to the special models, i.e., continuous-time MDPs and Markov chains, and can be extended to some simulation-based optimization methods such as reinforcement learning and neurodynamic programming, where estimation errors or approximation errors are common cases. Finally, we give an application example on the look-ahead control of a conveyor-serviced production station (CSPS), and show the corresponding error bounds.

The Estimation of the Semi-Markov Processes Performance Potentials Based on Parallel Simulation

Parallel Optimization for Markov Control Processes Based on Performance Potentials Simulation

Sensitivity Analysis and Performance Optimization of Semi-Markov Processes Based on Performance Potentials

Derivative estimates parallel simulation algorithm based on performance potentials theory

Hybrid Performance Modeling And Analyzing Of Parallel Systems

Two-Timescale Simulation-based Algorithm for Markov Decision Process Based on Performance Potentials

Simulation Optimization Algorithm for SMDPs with Parameterized Randomized Stationary Policies

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

Online Probabilistic Assessment of Operating Performance Based on Safety and Optimality Indices for Multimode Industrial Processes

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

On performance potentials and conditional Monte Carlo for gradient estimationfor Markov chains

Performance Optimization of Continuous-Time Markov Control Processes Based on Performance Potentials

Error bounds of optimization algorithms for semi-Markov decision processes

Sensitivity Estimates and Optimization Algorithms Based on Parallel Simulation for a Class of Closed Queuing Networks

Algorithm for Performance Sensitivity Estimation of Markov Discrete Event Dynamic System

Potential Based Optimization Algorithm Of Constrained Markov Decision Processes

A Simulation Optimization Algorithm for CTMDPs Based on Randomized Stationary Policies

Simulation-Based optimization of singularly perturbed markov reward processes with states aggregation

Simulation Estimates of Performance Sensitivity and Their Applications in Queuing Systems with Erlang Distributions

Optimization Algorithms for Semi-Markov Control Processes with Average Criteria

Optimal Stationary Policies for Semi-Markov Control Processes with Discounted-Cost Criteria