Abstract:Reinforcement learning has proved to be a powerful tool to solve optimal control problems over the past few years. However, the data-based constrained optimal control problem of nonaffine nonlinear discrete-time systems has rarely been studied yet. To solve this problem, an adaptive optimal control approach is developed by using the value iteration-based Q-learning (VIQL) with the critic-only structure. Most of the existing constrained control methods require the use of a certain performance index and only suit for linear or affine nonlinear systems, which is unreasonable in practice. To overcome this problem, the system transformation is first introduced with the general performance index. Then, the constrained optimal control problem is converted to an unconstrained optimal control problem. By introducing the action-state value function, i.e., Q-function, the VIQL algorithm is proposed to learn the optimal Q-function of the data-based unconstrained optimal control problem. The convergence results of the VIQL algorithm are established with an easy-to-realize initial condition . To implement the VIQL algorithm, the critic-only structure is developed, where only one neural network is required to approximate the Q-function. The converged Q-function obtained from the critic-only VIQL method is employed to design the adaptive constrained optimal controller based on the gradient descent scheme. Finally, the effectiveness of the developed adaptive control method is tested on three examples with computer simulation.

Learning-based adaptive optimal control of linear time-delay systems: A value iteration approach

A Learning-Based Optimal Tracking Controller for Continuous Linear Systems with Unknown Dynamics: Theory and Case Study

Robust Adaptive Iterative Learning Control for Discrete-Time Nonlinear Systems With Time-Iteration-Varying Parameters.

Value Iteration Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems

Optimal control for discrete-time affine non-linear systems using general value iteration

Discrete-Time Adaptive Iterative Learning Control for High-Order Nonlinear Systems with Unknown Control Directions

Neural-network-based Optimal Control for Discrete-Time Nonlinear Systems Using General Value Iteration

H ∞ optimal control of unknown linear systems by adaptive dynamic programming with applications to time‐delay systems

Learning‐based Model Predictive Control under Value Iteration with Finite Approximation Errors

Infinite-time stochastic linear quadratic optimal control for unknown discrete-time systems using adaptive dynamic programming approach

Adaptive Iterative Learning Control for Nonlinear Time-Delay Output Feedback Systems

Adaptive Constrained Optimal Control Design for Data-Based Nonlinear Discrete-Time Systems With Critic-Only Structure

Value Approximator-based Learning Model Predictive Control for Iterative Tasks

General value iteration based reinforcement learning for solving optimal tracking control problem of continuous-time affine nonlinear systems.

Linear Quadratic Tracking Control of Unknown Discrete-Time Systems Using Value Iteration Algorithm

Adaptive Output-Feedback Optimal Control for Continuous-Time Linear Systems Based on Adaptive Dynamic Programming Approach

Optimization-Based Learning Control for Nonlinear Time-Varying Systems

Value Iteration Based Integral Reinforcement Learning Approach for H∞ Controller Design of Continuous-Time Nonlinear Systems

Value Iteration-Based H∞ Controller Design for Continuous-Time Nonlinear Systems Subject to Input Constraints

Data-driven Adaptive Optimal Control for Discrete-Time Linear Time-Invariant Systems

An Adaptive Iterative Learning Control for Discrete-Time Nonlinear Systems with Iteration-Varying Uncertainties