Abstract:In this article, a real-time online off-policy reinforcement learning (RL) method is developed for the optimal control problem of unknown continuous-time nonlinear systems. First, by applying the temporal difference technique to the iterative procedure of off-policy RL, the iterative value function and the iterative policy input can be learned in real-time online. It is proven that the fitting error of neural network (NN) weights is exponentially convergent in each iteration. Second, a model-free Hamilton–Jacobi–Bellman equation (MF-HJBE) is deduced by taking the limit of the iterative procedure of off-policy RL. In this manner, it not only eliminates system dynamics in the classical HJBE, but also vanishes the iteration index. By applying temporal difference to the MF-HJBE, a real-time online tuning rule is designed to learn the optimal value function and the optimal policy input. It is proven that the fitting error of NN weights caused by the real-time online tuning rule is exponentially convergent. Note that the two online tuning rules, the iterative one and the real-time one, use only current and previous state data extracted from system trajectories. Meanwhile, it is proven using the Lyapunov's direct method that the system solution is uniformly ultimately bounded. Finally, simulation results demonstrate the validity of the proffered method.

Data-Driven Based Optimal Output-Feedback Control of Continuous-Time Systems

Output Feedback Adaptive Robust Learning Control of a Class of Nonlinear Systems with Periodic Disturbances

A Learning-Based Optimal Tracking Controller for Continuous Linear Systems with Unknown Dynamics: Theory and Case Study

Near Optimal Neural Network-based Output Feedback Control of Affine Nonlinear Discrete-Time Systems

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

An efficient data-based off-policy Q-learning algorithm for optimal output feedback control of linear systems

An Information-state based Approach to the Optimal Output Feedback Control of Nonlinear Systems

Reinforcement Learning-Based Control for a Class of Nonlinear Systems with Unknown Control Directions

Interactions of salts and denaturing agents with a polyacrylamide gel.

Robust Output Regulation and Reinforcement Learning-Based Output Tracking Design for Unknown Linear Discrete-Time Systems

Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems Using Online Approximators

Optimal dynamic output feedback control of unknown linear continuous-time systems by adaptive dynamic programming

State and Input Constrained Output-Feedback Adaptive Optimal Control of Affine Nonlinear Systems

Output Feedback Adaptive Optimal Control of Affine Nonlinear systems with a Linear Measurement Model

On Data-Driven Stochastic Output-Feedback Predictive Control

Data-Driven Deep Learning Based Feedback Linearization of Systems with Unknown Dynamics

The Adaptive Optimal Output Feedback Tracking Control of Unknown Discrete-Time Linear Systems Using a Multistep Q-Learning Approach

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Online Off-Policy Reinforcement Learning for Optimal Control of Unknown Nonlinear Systems Using Neural Networks

Model-based and Data-based Dynamic Output Feedback for Externally Positive Systems