Abstract:In this article, an adaptive dynamic programming (ADP) scheme utilizing a costate function is proposed for optimal control of unknown discrete-time nonlinear systems. The state-action data are obtained by interacting with the environment under the iterative scheme without any model information. In contrast with the traditional ADP scheme, the collected data in the proposed algorithm are generated with different policies, which improves data utilization in the learning process. In order to approximate the cost function more accurately and to achieve a better policy improvement direction in the case of insufficient data, a separate costate network is introduced to approximate the costate function under the actor-critic framework, and the costate is utilized as supplement information to estimate the cost function more precisely. Furthermore, convergence properties of the proposed algorithm are analyzed to demonstrate that the costate function plays a positive role in the convergence process of the cost function based on the alternate iteration mode of the costate function and cost function under a mild assumption. The uniformly ultimately bounded (UUB) property of all the variables is proven by using the Lyapunov approach. Finally, two numerical examples are presented to demonstrate the effectiveness and computation efficiency of the proposed method.

A New Self-Learning Optimal Control Laws for a Class of Discrete-Time Nonlinear Systems Based on ESN Architecture.

Discrete-Time Self-Learning Parallel Control

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Discrete-Time Adaptive Iterative Learning Control for High-Order Nonlinear Systems with Unknown Control Directions

Data-Driven Event-Triggered Adaptive Dynamic Programming Control for Nonlinear Systems with Input Saturation.

Control of Nonaffine Nonlinear Discrete-Time Systems Using Reinforcement-Learning-Based Linearly Parameterized Neural Networks

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Discrete-time adaptive iterative learning control with unknown control directions

Near Optimal Neural Network-based Output Feedback Control of Affine Nonlinear Discrete-Time Systems

Adaptive Iterative Learning Control for Discrete-Time Systems with Non-Repetitiveness

A New Approach to Finite-Horizon Optimal Control for Discrete-Time Affine Nonlinear Systems via a Pseudolinear Method

Online Adaptive Optimal Control for Continuous-Time Nonlinear Systems with Completely Unknown Dynamics.

Costate-Supplement ADP for Model-Free Optimal Control of Discrete-Time Nonlinear Systems

Intermittent Feedback Optimal Control of Saturated-Input Nonlinear Systems via Adaptive Dynamic Programming

Finite-time adaptive dynamic programming control for nonlinear systems based on novel complementary terminal sliding mode surface

Twin Deterministic Policy Gradient Adaptive Dynamic Programming for Optimal Control of Affine Nonlinear Discrete-time Systems

Self-Learning Optimal Control with Performance Analysis using Event-Triggered Adaptive Dynamic Programming

Approximately Optimal Control of Discrete-Time Nonlinear Switched Systems Using Globalized Dual Heuristic Programming

One-step-ahead optimal adaptive control based on ELS algorithm

Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces