Abstract:Reinforcement learning（RL） and adaptive/approximate dynamic programming（ADP） algorithms have recently received much attention from various scientific fields（e.g., artificial intelligence, systems and control, and applied mathematics）. This is partly due to their successful applications in a series of challenging problems, such as the sequential decision and optimal coordination control problems of large-scale multi-agent systems. In this paper, some preliminaries on RL and ADP algorithms are firstly introduced, and then the developments of these two closely related algorithms in different research fields are reviewed respectively, with emphasis on the developments from solving the sequential decision（optimal control） problem for single agent（control plant） to the sequential decision（optimal coordination control） problem of multi-agent systems by utilizing these two algorithms. Furthermore, after briefly surveying the structure evolution of the ADP algorithm in the last decades and the recent development of the ADP algorithm from model-based offline programming framework to model-free online learning framework, the research progress of the ADP algorithm in solving the optimal coordination control problem of multi-agent systems is reviewed. Finally, some interesting yet challenging issues on MARL algorithms and using ADP algorithms to solve optimal coordination control problem of multi-agent systems are suggested.

An Overview of Research on Adaptive Dynamic Programming

Adaptive dynamic programming: an introduction

Adaptive Dynamic Programming for Control: A Survey and Recent Advances

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Hamiltonian-Driven Adaptive Dynamic Programming With Approximation Errors

Approximate dynamic programming for continuous state and control problems

Finite-approximation-error-based Discrete-Time Iterative Adaptive Dynamic Programming.

Data-based Self-learning Optimal Control: Research Progress and Prospects

Hamiltonian-Driven Adaptive Dynamic Programming With Efficient Experience Replay

An Adaptive Dynamic Programming Algorithm to Solve Optimal Control of Uncertain Nonlinear Systems

Reinforcement learning and adaptive/approximate dynamic programming: A survey from theory to applications in multi-agent systems

A survey of adaptive optimal control theory

Adaptive Dynamic Programming for a Class of Complex-Valued Nonlinear Systems

Hamiltonian-driven Adaptive Dynamic Programming for Nonlinear Discrete-Time Dynamic Systems.

Discrete-Time Impulsive Adaptive Dynamic Programming.

Twin Deterministic Policy Gradient Adaptive Dynamic Programming for Optimal Control of Affine Nonlinear Discrete-time Systems

An Iterative Adaptive Dynamic Programming Algorithm for Optimal Control of Unknown Discrete-Time Nonlinear Systems with Constrained Inputs.

Online Approximate Optimal Control for Affine Non-Linear Systems with Unknown Internal Dynamics Using Adaptive Dynamic Programming

A Parallel Framework of Adaptive Dynamic Programming Algorithm with Off-Policy Learning.

Robust Optimal Control For Time-Delay Systems With Dynamic Uncertainties Via Adp

Robust adaptive dynamic programming for optimal nonlinear control design