Abstract:Cao's work shows that, by defining an α-dependent equivalent infinitesimal generator Aα, a semi-Markov decision process (SMDP) with both average- and discounted-cost criteria can be treated as an α-equivalent Markov decision process (MDP), and the performance potential theory can also be developed for SMDPs. In this work, we focus on establishing error bounds for potential and Aα-based iterative optimization methods. First, we introduce an α-uniformized Markov chain (UMC) for a SMDP via Aα and a uniformized parameter, and show their relations. Especially, we obtain that their performance potentials, as solutions of corresponding Poisson equations, are proportional, so that the studies of a SMDP and the α-UMC based on potentials are unified. Using these relations, we derive the error bounds for a potential-based policy-iteration algorithm and a value-iteration algorithm, respectively, when there exist various calculation errors. The obtained results can be applied directly to the special models, i.e., continuous-time MDPs and Markov chains, and can be extended to some simulation-based optimization methods such as reinforcement learning and neurodynamic programming, where estimation errors or approximation errors are common cases. Finally, we give an application example on the look-ahead control of a conveyor-serviced production station (CSPS), and show the corresponding error bounds.

Potential Based Optimization Algorithm Of Constrained Markov Decision Processes

Potentials Based Optimization with Embedded Markov Chain for Stochastic Constrained System

Constrained Optimization for Average Cost Continuous-Time Markov Decision Processes.

OPTIMALITY STRATEGY OF AVERAGE COST BASED PERFORMANCE POTENTIALS FOR MARKOV CONTROL PROCESS

Performance Optimization of Continuous-Time Markov Control Processes Based on Performance Potentials

Markov Decision Problems with Unbounded Transition Rates under Discounted-Cost Performance Criteria

Constrained Continuous-Time Markov Decision Processes with Average Criteria

Error bounds of optimization algorithms for semi-Markov decision processes

An Average Reward Performance Potential Estimation with Geometric Variance Reduction

Optimization Algorithms for Semi-Markov Control Processes with Average Criteria

Event-based optimization for finite-horizon total-cost markov decision processes

A Simulation Optimization Algorithm for CTMDPs Based on Randomized Stationary Policies

Performance Optimization Algorithms Based on Potentials for Semi-Markov Control Processes

On Average Optimality for Non-Stationary Markov Decision Processes in Borel Spaces

Linear Programming and Constrained Average Optimality for General Continuous-Time Markov Decision Processes in History-Dependent Policies.

Finding Optimal Observation-Based Policies for Constrained POMDPs under the Expected Average Reward Criterion

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

Simulation Optimization Algorithm for SMDPs with Parameterized Randomized Stationary Policies

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Average Optimality For Continuous-Time Markov Decision Processes In Polish Spaces

A unified approach for semi-Markov decision processes with discounted and average reward criteria