Abstract:Reinforcement learning has proved to be a powerful tool to solve optimal control problems over the past few years. However, the data-based constrained optimal control problem of nonaffine nonlinear discrete-time systems has rarely been studied yet. To solve this problem, an adaptive optimal control approach is developed by using the value iteration-based Q-learning (VIQL) with the critic-only structure. Most of the existing constrained control methods require the use of a certain performance index and only suit for linear or affine nonlinear systems, which is unreasonable in practice. To overcome this problem, the system transformation is first introduced with the general performance index. Then, the constrained optimal control problem is converted to an unconstrained optimal control problem. By introducing the action-state value function, i.e., Q-function, the VIQL algorithm is proposed to learn the optimal Q-function of the data-based unconstrained optimal control problem. The convergence results of the VIQL algorithm are established with an easy-to-realize initial condition . To implement the VIQL algorithm, the critic-only structure is developed, where only one neural network is required to approximate the Q-function. The converged Q-function obtained from the critic-only VIQL method is employed to design the adaptive constrained optimal controller based on the gradient descent scheme. Finally, the effectiveness of the developed adaptive control method is tested on three examples with computer simulation.

The intelligent critic framework for advanced optimal control

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Accelerated Optimization in Deep Learning with a Proportional-Integral-derivative Controller

On Modeling and Optimal Control of a Class of Hybrid Systems

Near Optimal Neural Network-based Output Feedback Control of Affine Nonlinear Discrete-Time Systems

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Adaptive Constrained Optimal Control Design for Data-Based Nonlinear Discrete-Time Systems With Critic-Only Structure

Kernel-based Adaptive Critic Designs for Optimal Control of Nonlinear Discrete-time System

Optimal Critic Learning for Robot Control in Time-Varying Environments.

Intelligent Optimal Control With Critic Learning for a Nonlinear Overhead Crane System

On Mixed Data and Event Driven Design for Adaptive-Critic-Based Nonlinear $h_{\infty}$ Control

Solver-Critic: A Reinforcement Learning Method for Discrete-Time-Constrained-Input Systems

Optimal H∞ Tracking Control of Nonlinear Systems with Zero-Equilibrium-free Via Novel Adaptive Critic Designs.

Event-triggered optimal control with finite-time convergence critic networks for input-constrained nonlinear systems

Constrained Online Optimal Control for Continuous-Time Nonlinear Systems Using Neuro-Dynamic Programming

Intelligent Optimized Control: Overview and Prospect

Robust Adaptive Critic Control Design with Network-Based Event-Triggered Formulation

On Mixed Data and Event Driven Design for Adaptive-Critic-Based Nonlinear $H_{\infty}$ Control.

Indirect Adaptive Fuzzy-Regulated Optimal Control for Unknown Continuous-Time Nonlinear Systems.

Goal representation adaptive critic design for discrete-time uncertain systems subjected to input constraints: The event-triggered case

Thoughts on Intelligent Control