Abstract:In this article, a real-time online off-policy reinforcement learning (RL) method is developed for the optimal control problem of unknown continuous-time nonlinear systems. First, by applying the temporal difference technique to the iterative procedure of off-policy RL, the iterative value function and the iterative policy input can be learned in real-time online. It is proven that the fitting error of neural network (NN) weights is exponentially convergent in each iteration. Second, a model-free Hamilton–Jacobi–Bellman equation (MF-HJBE) is deduced by taking the limit of the iterative procedure of off-policy RL. In this manner, it not only eliminates system dynamics in the classical HJBE, but also vanishes the iteration index. By applying temporal difference to the MF-HJBE, a real-time online tuning rule is designed to learn the optimal value function and the optimal policy input. It is proven that the fitting error of NN weights caused by the real-time online tuning rule is exponentially convergent. Note that the two online tuning rules, the iterative one and the real-time one, use only current and previous state data extracted from system trajectories. Meanwhile, it is proven using the Lyapunov's direct method that the system solution is uniformly ultimately bounded. Finally, simulation results demonstrate the validity of the proffered method.

Self-learning Optimal Control for Uncertain Nonlinear Systems Via Online Updated Cost Function

Neural-Network-Based Online Optimal Control for Uncertain Non-Linear Continuous-Time Systems with Control Constraints

A learning optimal control scheme for robust stabilization of a class of uncertain nonlinear systems

A neural-network-based online optimal control approach for nonlinear robust decentralized stabilization

A Novel Self-Learning Optimal Control Approach for Decentralized Guaranteed Cost Control of A Class of Complex Nonlinear Systems

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Online reinforcement learning control of unknown nonaffine nonlinear discrete time systems

Neural-Network-Based Online HJB Solution for Optimal Robust Guaranteed Cost Control of Continuous-Time Uncertain Nonlinear Systems

Reinforcement Learning-Based Optimal Stabilization for Unknown Nonlinear Systems Subject to Inputs With Uncertain Constraints

Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems Using Online Approximators

Data-Based Self-Learning Optimal Control For Continuous-Time Unknown Nonlinear Systems With Disturbance

Online Adaptive Optimal Control for Continuous-Time Nonlinear Systems with Completely Unknown Dynamics.

Adaptive Critic Learning-Based Robust Control of Systems with Uncertain Dynamics

Self-learning Robust Optimal Control for Continuous-Time Nonlinear Systems with Mismatched Disturbances

Adaptive Optimal Control for a Class of Continuous-Time Affine Nonlinear Systems with Unknown Internal Dynamics

Adaptive Neural Inverse Optimal Tracking Control for Uncertain Nonlinear System with Actuator Faults

Near-optimal Online Control of Uncertain Nonlinear Continuous-Time Systems Based on Concurrent Learning.

Robust Tracking Control of Uncertain Nonlinear Systems with Adaptive Dynamic Programming

Online Approximate Optimal Control for Affine Non-Linear Systems with Unknown Internal Dynamics Using Adaptive Dynamic Programming

Online Synchronous Iterative Algorithm for Optimal Control of Stochastic Affine Nonlinear Systems

Online Off-Policy Reinforcement Learning for Optimal Control of Unknown Nonlinear Systems Using Neural Networks