Abstract:The optimization problems of Markov control processes (MCPs) with exact knowledge of system parameters, in the form of transition probabilities or infinitesimal transition rates, can be solved by using the concept of Markov performance potential which plays an important role in the sensitivity analysis of MCPs. In this paper, by using an equivalent infinitesimal generator, we first introduce a definition of discounted Poisson equations for semi-Markov control processes (SMCPs), which is similar to that for MCPs, and the performance potentials of SMCPs are defined as solution of the equation. Some related optimization techniques based on performance potentials for MCPs may be extended to the optimization of SMCPs if the system parameters are known with certainty. Unfortunately, exact values of the distributions of the sojourn times at some states or the transition probabilities of the embedded Markov chain for a large-scale SMCP are generally difficult or impossible to obtain, which leads to the uncertainty of the semi-Markov kernel, and thereby to the uncertainty of equivalent infinitesimal transition rates. Similar to the optimization of uncertain MCPs, a potential-based policy iteration method is proposed in this work to search for the optimal robust control policy for SMCPs with uncertain infinitesimal transition rates that are represented as compact sets. In addition, convergence of the algorithm is discussed.

Optimal Policies for a Continuous Time MCP with Compact Action Set

Optimal Stationary Policies for a Class of Countable Markov Control Processes

Performance Optimization of Continuous-Time Markov Control Processes Based on Performance Potentials

Optimization Algorithms for Semi-Markov Control Processes with Average Criteria

The Optimal Robust Control Policy for Uncertain Semi-Markov Control Processes

A Simulation Optimization Algorithm for CTMDPs Based on Randomized Stationary Policies

Optimal Stationary Policies for Semi-Markov Control Processes with Discounted-Cost Criteria

On the Convergence of Optimal Actions for Markov Decision Processes and the Optimality of $(s,S)$ Inventory Policies

Optimal Time-Abstract Schedulers for CTMDPs and Markov Games

Average-Cost MDPs with Infinite State and Action Sets: New Sufficient Conditions for Optimality Inequalities and Equations

Continuous-Time Policy Optimization.

The minimal hitting probability of continuous-time controlled Markov systems with countable states

Sufficiency of Markov Policies for Continuous-Time Jump Markov Decision Processes

ON GRADUAL-IMPULSE CONTROL OF CONTINUOUS-TIME MARKOV DECISION PROCESSES WITH EXPONENTIAL UTILITY

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Policy-Iteration-Based Finite-Horizon Approximate Dynamic Programming for Continuous-Time Nonlinear Optimal Control

Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action

A New Continuous-Time Policy Iteration for Time-Varying Nonlinear Systems

Asymptotically Optimal Policies for Weakly Coupled Markov Decision Processes

On the optimality equation for average cost Markov decision processes and its validity for inventory control

Anytime-Competitive Reinforcement Learning with Policy Prior