Abstract:In this paper, a convex optimization-based method is proposed for numerically solving dynamic programs in continuous state and action spaces. The key idea is to approximate the output of the Bellman operator at a particular state by the optimal value of a convex program. The approximate Bellman operator has a computational advantage because it involves a convex optimization problem in the case of control-affine systems and convex costs. Using this feature, we propose a simple dynamic programming algorithm to evaluate the approximate value function at pre-specified grid points by solving convex optimization problems in each iteration. We show that the proposed method approximates the optimal value function with a uniform convergence property in the case of convex optimal value functions. We also propose an interpolation-free design method for a control policy, of which performance converges uniformly to the optimum as the grid resolution becomes finer. When a nonlinear control-affine system is considered, the convex optimization approach provides an approximate policy with a provable suboptimality bound. For general cases, the proposed convex formulation of dynamic programming operators can be modified as a nonconvex bi-level program, in which the inner problem is a linear program, without losing uniform convergence properties.

Elementary results on solutions to the bellman equation of dynamic programming: existence, uniqueness, and convergence

Existence and uniqueness of solutions to the Bellman equation in stochastic dynamic programming

Existence and Iterative Approximations of Solutions for Certain Functional Equation and Inequality

On the fragility of the basis on the Hamilton–Jacobi–Bellman equation in economic dynamics

Do not Blame Bellman: It Is Koopmans' Fault

Convergence of Newton'S Method and Uniqueness of the Solution of Equations in Banach Spaces Ii

Convergence of Dynamic Programming Models

An approximation approach to dynamic programming with unbounded returns

An elementary proof of the existence and uniqueness of solutions to an initial value problem

Existence and Uniqueness of Solutions to the Stochastic Bellman Equation with Unbounded Shock

On the Existence of Solutions to the Dynamic User Equilibrium Problem.

Numerical Solutions to the Bellman Equation of Optimal Control

Coase Meets Bellman: Dynamic Programming for Production Networks

Some existence theorems of common and coincidence solutions for a class of systems of functional equations arising in dynamic programming

On the uniqueness of solutions to quadratic BSDEs with non-convex generators and unbounded terminal conditions: the certain exponential moment case

Newton’s Method, Bellman Recursion and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Unbounded Markov Dynamic Programming with Weighted Supremum Norm Perov Contractions

On the existence and uniqueness of unbounded solutions to quadratic BSDEs with monotonic-convex generators

Gradient-Bounded Dynamic Programming with Submodular and Concave Extensible Value Functions

A Convex Optimization Approach to Dynamic Programming in Continuous State and Action Spaces

Long time asymptotics for fully nonlinear Bellman equations: A backward SDE approach