Abstract:We consider inexact policy iteration methods for large-scale infinite-horizon discounted MDPs with finite spaces, a variant of policy iteration where the policy evaluation step is implemented inexactly using an iterative solver for linear systems. In the classical dynamic programming literature, a similar principle is deployed in optimistic policy iteration, where an a-priori fixed-number of iterations of value iteration is used to inexactly solve the policy evaluation step. Inspired by the connection between policy iteration and semismooth Newton's method, we investigate a class of iPI methods that mimic the inexact variants of semismooth Newton's method by adopting a parametric stopping condition to regulate the level of inexactness of the policy evaluation step. For this class of methods we discuss local and global convergence properties and derive a practical range of values for the stopping-condition parameter that provide contraction guarantees. Our analysis is general and therefore encompasses a variety of iterative solvers for policy evaluation, including the standard value iteration as well as more sophisticated ones such as GMRES. As underlined by our analysis, the selection of the inner solver is of fundamental importance for the performance of the overall method. We therefore consider different iterative methods to solve the policy evaluation step and analyze their applicability and contraction properties when used for policy evaluation. We show that the contraction properties of these methods tend to be enhanced by the specific structure of policy evaluation and that there is margin for substantial improvement in terms of convergence rate. Finally, we study the numerical performance of different instances of inexact policy iteration on large-scale MDPs for the design of health policies to control the spread of infectious diseases in epidemiology.

Solving Linear DSGE Models with Bernoulli Iterations

Solving linear DSGE models with Newton methods

A General and Efficient Method for Solving Regime-Switching DSGE Models

Analyzing Linear DSGE models: the Method of Undetermined Markov States

Computing Dsge Models with Recursive Preferences and Stochastic Volatility

Computing DSGE Models with Recursive Preferences

A Nonparametric GMM Series Approach to Solving Multi-equation Asset Pricing Models with Recursive Preferences

Semi-Global Solutions to DSGE Models: Perturbation around a Deterministic Path

Empirical Macroeconomics and DSGE Modeling in Statistical Perspective

Solving Linear Difference Systems with Lagged Expectations by a Method of Undetermined Coefficients

Global DSGE Models

Non-linear approximations of DSGE models with neural-networks and hard-constraints

Three Improvements to Multi-Level Monte Carlo Simulation of SDE Systems

On the Low Degree of Entropy Implied by the Solutions of Modern Macroeconomic Models

Inexact Policy Iteration Methods for Large-Scale Markov Decision Processes

Comment on An and Schorfheide's Bayesian Analysis of DSGE Models

Efficient Likelihood-based Estimation Via Annealing for Dynamic Structural Macrofinance Models

A segment-wise dynamic programming algorithm for BSDEs

A Study of the Finite Sample Properties of EMM, GMM, QMLE, and MLE for a Square-Root Interest Rate Diffusion Model

A DSGE Consumption Function in a CGE Model: Parameter Estimation by CGE Simulation

Perturbating and Estimating DSGE Models in Julia