Abstract:In this article, a real-time online off-policy reinforcement learning (RL) method is developed for the optimal control problem of unknown continuous-time nonlinear systems. First, by applying the temporal difference technique to the iterative procedure of off-policy RL, the iterative value function and the iterative policy input can be learned in real-time online. It is proven that the fitting error of neural network (NN) weights is exponentially convergent in each iteration. Second, a model-free Hamilton–Jacobi–Bellman equation (MF-HJBE) is deduced by taking the limit of the iterative procedure of off-policy RL. In this manner, it not only eliminates system dynamics in the classical HJBE, but also vanishes the iteration index. By applying temporal difference to the MF-HJBE, a real-time online tuning rule is designed to learn the optimal value function and the optimal policy input. It is proven that the fitting error of NN weights caused by the real-time online tuning rule is exponentially convergent. Note that the two online tuning rules, the iterative one and the real-time one, use only current and previous state data extracted from system trajectories. Meanwhile, it is proven using the Lyapunov's direct method that the system solution is uniformly ultimately bounded. Finally, simulation results demonstrate the validity of the proffered method.

Reinforcement Learning with Adjustable Convergence Rate for Data-Based Nonlinear Control

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Nonlinear Neuro-Optimal Tracking Control via Stable Iterative Q-Learning Algorithm

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Online Off-Policy Reinforcement Learning for Optimal Control of Unknown Nonlinear Systems Using Neural Networks

Reinforcement Learning-Based Control for a Class of Nonlinear Systems with Unknown Control Directions

Q-learning for Optimal Control of Continuous-time Systems

Reinforcement Learning-Based Direct Adaptive Optimal Control of JLQ Model

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems Using Online Approximators

Generalized Policy Iteration-based Reinforcement Learning Algorithm for Optimal Control of Unknown Discrete-time Systems

A Combined Policy Gradient and Q-learning Method for Data-driven Optimal Control Problems

Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators

An efficient data-based off-policy Q-learning algorithm for optimal output feedback control of linear systems

Hybrid Reinforcement Learning for Optimal Control of Non-Linear Switching System

Discrete-time optimal control scheme based on Q-learning algorithm

Policy Gradient Reinforcement Learning for Parameterized Continuous-Time Optimal Control

Adaptive Optimal Control of Discrete-Time Linear Systems with Discounted Value: Off-Policy Reinforcement Learning

Iterative Q-learning-based Nonlinear Optimal Tracking Control

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

Adaptive Optimal Control with Guaranteed Convergence Rate for Continuous-Time Linear Systems with Completely Unknown Dynamics.