Abstract:Policy iteration and value iteration are two main iterative adaptive dynamic programming frameworks for solving optimal control problems. Policy iteration converges fast while requiring an initial stabilizing control policy, which is a strict constraint in practice. Value iteration avoids the requirement of initial admissible control policy while converging much slowly. This paper tries to utilize the advantages of policy iteration and value iteration, and avoids their drawbacks at the same time. Therefore, a multi-step heuristic dynamic programming (MsHDP) method is developed for solving the optimal control problem of nonlinear discrete-time systems. MsHDP speeds up value iteration and avoids the requirement of initial admissible control policy in policy iteration at the same time. The convergence theory of MsHDP is established by proving that it converges to the solution of the Bellman equation. For implementation purpose, the actor-critic neural network (NN) structure is developed. The critic NN is employed to estimate the value function and its NN weight vector is computed with a least-square scheme. The actor NN is used to estimate the control policy and a gradient descent method is proposed for updating its NN weight vector. According to the comparative simulation studies on two examples, the effectiveness and advantages of MsHDP are verified.

Modified general policy iteration based adaptive dynamic programming for unknown discrete‐time linear systems

Modified λ-Policy Iteration Based Adaptive Dynamic Programming for Unknown Discrete-Time Linear Systems

Modified $\lambda$-Policy Iteration Based Adaptive Dynamic Programming for Unknown Discrete-Time Linear Systems

Discrete-Time Nonlinear Generalized Policy Iteration for Optimal Control Using Neural Networks

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Local Policy Iteration Adaptive Dynamic Programming for Discrete-Time Nonlinear Systems

Bias-Policy Iteration-Based Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems

Policy Gradient Adaptive Dynamic Programming for Data-Based Optimal Control.

Discrete-Time Stable Generalized Self-Learning Optimal Control With Approximation Errors.

Bias-policy iteration based adaptive dynamic programming for unknown continuous-time linear systems

Twin Deterministic Policy Gradient Adaptive Dynamic Programming for Optimal Control of Affine Nonlinear Discrete-time Systems

Discrete-Time Optimal Control Via Local Policy Iteration Adaptive Dynamic Programming

Policy Optimization Adaptive Dynamic Programming for Optimal Control of Input-Affine Discrete-Time Nonlinear Systems.

Deterministic Policy Gradient Adaptive Dynamic Programming for Model-Free Optimal Control

A Generalized Policy Iteration Adaptive Dynamic Programming Algorithm for Optimal Control of Discrete-Time Nonlinear Systems with Actuator Saturation.

Adaptive dynamic programming-based algorithm for infinite-horizon linear quadratic stochastic optimal control problems

Policy Gradient Adaptive Dynamic Programming for Nonlinear Discrete-Time Zero-Sum Games with Unknown Dynamics

Multi-step Heuristic Dynamic Programming for Optimal Control of Nonlinear Discrete-Time Systems.

On policy iteration‐based discounted optimal control

Infinite Horizon Self-Learning Optimal Control of Nonaffine Discrete-Time Nonlinear Systems

Discrete-time Generalized Policy Iteration ADP Algorithm with Approximation Errors.