Abstract:Dynamic programming (DP) is an approach to computing the optimal control policy over time under nonlinearity and uncertainty by employing the principle of optimality introduced by Richard Bellman. Instead of enumerating all possible control sequences, dynamic programming only searches admissible state and/or action values that satisfy the principle of optimality. Therefore, the computation complexity can be much improved over the direct enumeration method. However, the computational efforts and the data storage requirement increase exponentially with the dimensionality of the system, which are reflected in the three curses: the state space, the observation space, and the action space. Thus, the traditional DP approach was limited to solving small size problems. This paper aims at providing an overview of latest development of a class of approximate/adaptive dynamic programming algorithms including those applicable to continuous state and continuous control problems. The paper will especially review direct heuristic dynamic programming (direct (HDP), its design and applications, which include large and complex continuous state and control problems. In addition to the basic principle of direct HDP, the paper includes two application studies of the direct HDP - one is when it is used in a nonlinear tracking problem, and the other is on a power grid coordination control problem based on China southern network.

Convergence of Dynamic Programming Models

A Convex Optimization Approach to Dynamic Programming in Continuous State and Action Spaces

Revisiting approximate dynamic programming and its convergence

Gradient-Bounded Dynamic Programming with Submodular and Concave Extensible Value Functions

Convergence Guarantee of Dynamic Programming for LTL Surrogate Reward

Dynamic Programming: Optimality at a Point Implies Optimality Everywhere

Convergence of Markov Decision Processes with Constraints and State-Action Dependent Discount Factors

Approximate dynamic programming for continuous state and control problems

Approximate Constrained Discounted Dynamic Programming with Uniform Feasibility and Optimality

Approximation Algorithms for Optimization of Combinatorial Dynamical Systems

An approximation approach to dynamic programming with unbounded returns

Convergence of Controlled Models and Finite-State Approximation for Discounted Continuous-Time Markov Decision Processes with Constraints

Discrete Approximation and Convergence Analysis for a Class of Decision-Dependent Two-Stage Stochastic Linear Programs

The Principle of Optimality in Dynamic Programming: A Pedagogical Note

Improved Dynamic Programming Methods for Optimal Control of Lumped-Parameter Stochastic Systems

Convergence of Expectation-Maximization Algorithm With Mixed-Integer Optimization

Complexity of stochastic dual dynamic programming

Convergence of an Algorithm for Optimal Control of Nonlinear Continuous-Time Systems with Model-Reality Differences

Converging outer approximations to global attractors using semidefinite programming

Rates of Convergence in Certain Native Spaces of Approximations used in Reinforcement Learning