Abstract:We consider the infinite-horizon discounted optimal control problem formalized by Markov Decision Processes. We focus on several approximate variations of the Policy Iteration algorithm: Approximate Policy Iteration, Conservative Policy Iteration (CPI), a natural adaptation of the Policy Search by Dynamic Programming algorithm to the infinite-horizon case (PSDP$_\infty$), and the recently proposed Non-Stationary Policy iteration (NSPI(m)). For all algorithms, we describe performance bounds, and make a comparison by paying a particular attention to the concentrability constants involved, the number of iterations and the memory required. Our analysis highlights the following points: 1) The performance guarantee of CPI can be arbitrarily better than that of API/API($\alpha$), but this comes at the cost of a relative---exponential in $\frac{1}{\epsilon}$---increase of the number of iterations. 2) PSDP$_\infty$ enjoys the best of both worlds: its performance guarantee is similar to that of CPI, but within a number of iterations similar to that of API. 3) Contrary to API that requires a constant memory, the memory needed by CPI and PSDP$_\infty$ is proportional to their number of iterations, which may be problematic when the discount factor $\gamma$ is close to 1 or the approximation error $\epsilon$ is close to $0$; we show that the NSPI(m) algorithm allows to make an overall trade-off between memory and performance. Simulations with these schemes confirm our analysis.

Policy Iteration for Continuous-Time Average Reward Markov Decision Processes in Polish Spaces

Average Optimality For Continuous-Time Markov Decision Processes In Polish Spaces

Continuous-Time Markov Decision Processes with Discounted Rewards: the Case of Polish Spaces

Discounted Optimality for Continuous-Time Markov Decision Processes in Polish Spaces

Continuous-Time Markov Decision Processes with State-Dependent Discount Factors

Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equations

Continuous-Time Markov Decision Processes with Unbounded Transition and Discounted-Reward Rates

New Discount and Average Optimality Conditions for Continuous-Time Markov Decision Processes

Optimality of Mixed Policies for Average Continuous-Time Markov Decision Processes with Constraints

Constrained Semi-Markov Decision Processes with Ratio and Time Expected Average Criteria in Polish Spaces

Finding Optimal Memoryless Policies of POMDPs under the Expected Average Reward Criterion

Linear Programming and Constrained Average Optimality for General Continuous-Time Markov Decision Processes in History-Dependent Policies.

Discounted Continuous-Time Constrained Markov Decision Processes in Polish Spaces

First Passage Optimality for Continuous-Time Markov Decision Processes with Varying Discount Factors and History-Dependent Policies

On Convergence Analysis of Policy Iteration Algorithms for Entropy-Regularized Stochastic Control Problems

Value Iteration For Average Cost Markov Decision Processes In Borel Spaces

Policy iteration for parameterized Markov decision processes and its application

New optimality conditions for average-payoff continuous-time Markov games in Polish spaces

Approximate Policy Iteration Schemes: A Comparison

A note on the policy iteration algorithm for discounted Markov decision processes for a class of semicontinuous models

Optimal Control of Ergodic Continuous-Time Markov Chains with Average Sample-Path Rewards