Abstract:In this article, a model-free online adaptive dynamic programming (ADP) approach is developed for solving the optimal control problem of nonaffine nonlinear systems. Combining the off-policy learning mechanism with the parallel paradigm, multithread agents are employed to collect the transitions by interacting with the environment that significantly augments the number of sampled data. On the other hand, each thread agent explores the environment with different initial states under its own behavior policy that enhances the exploration capability and alleviates the correlation between the sampled data. After the policy evaluation process, only one step update is required for policy improvement based on the policy gradient method. The stability of the system under iterative control laws is guaranteed. Moreover, the convergence analysis is given to prove that the iterative Q-function is monotonically nonincreasing and finally converges to the solution of the Hamilton-Jacobi-Bellman (HJB) equation. For implementing the algorithm, the actor-critic (AC) structure is utilized with two neural networks (NNs) to approximate the Q-function and the control policy. Finally, the effectiveness of the proposed algorithm is verified by two numerical examples.

Research of a Parallel Learning Adaptive Dynamic Programming Based on Genetic Algorithms

Parallel adaptive hybrid genetic optimization algorithm and its application

Adaptive Dynamic Programming Based on Parallel Control Theory for Underwater Vehicles.

A Parallel Framework of Adaptive Dynamic Programming Algorithm with Off-Policy Learning.

A CUDA-based Parallel Adaptive Dynamic Programming Algorithm

An Overview of Research on Adaptive Dynamic Programming

A kind of utility function in adaptive dynamic programming for inverted pendulum control

Discrete-Time Self-Learning Parallel Control

Online Sparse Kernel Learning-Based Adaptive Dynamic Programming

Adaptive Primal-Dual Genetic Algorithms in Dynamic Environments.

Twin Deterministic Policy Gradient Adaptive Dynamic Programming for Optimal Control of Affine Nonlinear Discrete-time Systems

Iterative ADP Learning Algorithms for Discrete-Time Multi-Player Games.

Neural-network-based Learning Algorithms for Cooperative Games of Discrete-Time Multi-Player Systems with Control Constraints Via Adaptive Dynamic Programming

Adaptive Dynamic Programming for Control: A Survey and Recent Advances

A Summary on Some Typical Adaptive Dynamic Programming Schemes

Adaptive dynamic programming: an introduction

Finite-approximation-error-based Discrete-Time Iterative Adaptive Dynamic Programming.

Policy Gradient Adaptive Dynamic Programming for Data-Based Optimal Control.

Heuristic Dynamic Programming Strategy with Eligibility Traces

Online Optimal Control of Discrete-Time Systems Based on Globalized Dual Heuristic Programming with Eligibility Traces

On Distributed Implementation of Switch-Based Adaptive Dynamic Programming