Abstract:The optimization problems of Markov control processes (MCPs) with exact knowledge of system parameters, in the form of transition probabilities or infinitesimal transition rates, can be solved by using the concept of Markov performance potential which plays an important role in the sensitivity analysis of MCPs. In this paper, by using an equivalent infinitesimal generator, we first introduce a definition of discounted Poisson equations for semi-Markov control processes (SMCPs), which is similar to that for MCPs, and the performance potentials of SMCPs are defined as solution of the equation. Some related optimization techniques based on performance potentials for MCPs may be extended to the optimization of SMCPs if the system parameters are known with certainty. Unfortunately, exact values of the distributions of the sojourn times at some states or the transition probabilities of the embedded Markov chain for a large-scale SMCP are generally difficult or impossible to obtain, which leads to the uncertainty of the semi-Markov kernel, and thereby to the uncertainty of equivalent infinitesimal transition rates. Similar to the optimization of uncertain MCPs, a potential-based policy iteration method is proposed in this work to search for the optimal robust control policy for SMCPs with uncertain infinitesimal transition rates that are represented as compact sets. In addition, convergence of the algorithm is discussed.

Markov Decision Problems with Unbounded Transition Rates under Discounted-Cost Performance Criteria

Performance Optimization for Countable Semi-Markov Decision Processes with Discounted-cost

Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria.

Optimal Stationary Policies for Semi-Markov Control Processes with Discounted-Cost Criteria

Performance Optimization of Continuous-Time Markov Control Processes Based on Performance Potentials

Optimal Stationary Policies for a Class of Countable Markov Control Processes

Error bounds of optimization algorithms for semi-Markov decision processes

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

OPTIMALITY STRATEGY OF AVERAGE COST BASED PERFORMANCE POTENTIALS FOR MARKOV CONTROL PROCESS

Mixed Markov Decision Processes in a Semi-Markov Environment with Discounted Criterion

Continuous Time Markov Decision Processes with Discounted Moment Criterion

A survey of recent results on continuous-time Markov decision processes

Markov Decision Processes under Risk Sensitivity: A Discount Vanishing Approach

Constrained Markov Decision Processes with Non-constant Discount Factor

Potential Based Optimization Algorithm Of Constrained Markov Decision Processes

Performance Optimization Algorithms Based on Potentials for Semi-Markov Control Processes

Markov Decision Processes with Time-Varying Geometric Discounting

The Optimal Robust Control Policy for Uncertain Semi-Markov Control Processes

Continuous Time Markov Decision Processes with Nonuniformly Bounded Transition Rate: Expected Total Rewards

Relations Between Discounted Models and Average Models for Semi-Markov Decision Processes

A Simulation Optimization Algorithm for CTMDPs Based on Randomized Stationary Policies