Abstract:Continuous-time nonlinear optimal control problems hold great promise in real-world applications. After decades of development, reinforcement learning (RL) has achieved some of the greatest successes as a general nonlinear control design method. However, a recent comprehensive analysis of state-of-the-art continuous-time RL (CT-RL) methods, namely, adaptive dynamic programming (ADP)-based CT-RL algorithms, reveals they face significant design challenges due to their complexity, numerical conditioning, and dimensional scaling issues. Despite advanced theoretical results, existing ADP CT-RL synthesis methods are inadequate in solving even small, academic problems. The goal of this work is thus to introduce a suite of new CT-RL algorithms for control of affine nonlinear systems. Our design approach relies on two important factors. First, our methods are applicable to physical systems that can be partitioned into smaller subproblems. This constructive consideration results in reduced dimensionality and greatly improved intuitiveness of design. Second, we introduce a new excitation framework to improve persistence of excitation (PE) and numerical conditioning performance via classical input/output insights. Such a design-centric approach is the first of its kind in the ADP CT-RL community. In this paper, we progressively introduce a suite of (decentralized) excitable integral reinforcement learning (EIRL) algorithms. We provide convergence and closed-loop stability guarantees, and we demonstrate these guarantees on a significant application problem of controlling an unstable, nonminimum phase hypersonic vehicle (HSV).

Data-based neural controls for an unknown continuous-time multi-input system with integral reinforcement

Robust optimal control of the multi‐input systems with unknown disturbance based on adaptive integral reinforcement learning Q‐function

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

A Learning-Based Optimal Tracking Controller for Continuous Linear Systems with Unknown Dynamics: Theory and Case Study

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Reinforcement Learning-Based Control for a Class of Nonlinear Systems with Unknown Control Directions

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Control of Nonaffine Nonlinear Discrete-Time Systems Using Reinforcement-Learning-Based Linearly Parameterized Neural Networks

Robust Near-optimal Control for Constrained Nonlinear System via Integral Reinforcement Learning

Online reinforcement learning control of unknown nonaffine nonlinear discrete time systems

Novel iterative neural dynamic programming for data-based approximate optimal control design

Predictive Control of Voltage Source Inverter: an Online Reinforcement Learning Solution

Online Off-Policy Reinforcement Learning for Optimal Control of Unknown Nonlinear Systems Using Neural Networks

Continuous-Time Reinforcement Learning: New Design Algorithms with Theoretical Insights and Performance Guarantees

Optimal Asymptotic Tracking Control for Nonzero-Sum Differential Game Systems with Unknown Drift Dynamics via Integral Reinforcement Learning

Adaptive Neural Network Control of Nonlinear Systems with Unknown Dynamics

Decentralized Adaptive Neural Inverse Optimal Control of Nonlinear Interconnected Systems

Off-Policy Reinforcement Learning for $ H_\infty $ Control Design

Integral reinforcement learning‐based optimal tracking control for uncertain nonlinear systems under input constraint and specified performance constraints

Neural network-based finite-horizon optimal control of uncertain affine nonlinear discrete-time systems

Output information-based intermittent optimal control for continuous-time nonlinear systems with unmatched uncertainties via adaptive dynamic programming