Abstract:This paper presents a new formulation for model-free robust optimal regulation of continuous-time nonlinear systems. The proposed reinforcement learning based approach, referred to as incremental adaptive dynamic programming (IADP), exploits measured data to allow the design of the approximate optimal incremental control strategy, which stabilizes the controlled system incrementally under model uncertainties, environmental disturbances, and input saturation. By leveraging the time delay estimation (TDE) technique, we first exploit sensory data to reduce the requirement of a complete dynamics, where measured data are adopted to construct an incremental dynamics that reflects the system evolution in an incremental form. Then, the resulting incremental dynamics serves to design the approximate optimal incremental control strategy based on adaptive dynamic programming, which is implemented as a simplified single critic structure to get the approximate solution to the value function of the Hamilton-Jacobi-Bellman equation. Furthermore, for the critic artificial neural network, experience data are used to design an off-policy weight update law with guaranteed weight convergence. Rather importantly, to address the unintentionally introduced TDE error, we incorporate a TDE error bound related term into the cost function, whereby the TDE error is attenuated during the optimization process. The system stability proof and the weight convergence proof are provided. Numerical simulations are conducted to validate the effectiveness and superiority of our proposed IADP, especially regarding the reduced control energy expenditure and the enhanced robustness.

Efficient model-based reinforcement learning for approximate online optimal

Model-based reinforcement learning for infinite-horizon approximate optimal tracking

Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems Using Online Approximators

Kernel-Based Approximate Dynamic Programming for Real-Time Online Learning Control: An Experimental Study

Near Optimal Control for a Class of Stochastic Hybrid Systems.

Model-Free Incremental Adaptive Dynamic Programming Based Approximate Robust Optimal Regulation

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Online reinforcement learning control of unknown nonaffine nonlinear discrete time systems

Online adaptive learning of optimal control solutions using integral reinforcement learning

State and Parameter Estimation for Affine Nonlinear Systems

An Adaptive Cross Approximation-based Method for Robust Nonlinear Feedback Control Problems

Data-based reinforcement learning approximate optimal control for an uncertain nonlinear system with control effectiveness faults

Online Synchronous Iterative Algorithm for Optimal Control of Stochastic Affine Nonlinear Systems

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Sparse Online Kernelized Actor-Critic Learning in Reproducing Kernel Hilbert Space

Data-Driven Near-Optimal Control of Nonlinear Systems Over Finite Horizon

Average Cost Optimal Control of Stochastic Systems Using Reinforcement Learning

Online Off-Policy Reinforcement Learning for Optimal Control of Unknown Nonlinear Systems Using Neural Networks

Online Stackelberg Optimization via Nonlinear Control

Adaptive Optimal Control for a Class of Continuous-Time Affine Nonlinear Systems with Unknown Internal Dynamics

An Online Algorithm for Solving Feedback Optimal Control Problems with Partial Observations