Abstract:In this paper, to tackle the optimal control problem, we propose a $\mathcal{Q}$ -Learning approach for continuous-time nonlinear systems without any dynamic information. Primarily, the Hamiltonian and optimum cost functions are utilized to articulate the $\mathcal{Q}$ -function of continuous-time affine systems. To reduce the dependence of algorithms on system information, a novel $\mathcal{Q}$ -Learning approach is derived to obtain optimal solutions of nonlinear continuous-time systems without requiring knowledge of either the drift information $p(x)$ or input gain $q(x)$ . To implement this approach, critic and actor neural networks can be iterated alternately using an integral reinforcement learning method to estimate the $\mathcal{Q}$ -function. Furthermore, all signals in closed-loop system are demonstrated to be ultimate uniform bounded (UUB). It is worth noting that there exist rare literatures focused on the optimal control problem of continuous-time nonlinear uncertain systems via the $\mathcal{Q}$ -Learning for actor/critic networks iteration. Finally, two simulations are used to confirm the effectiveness of the proposed algorithm. Note to Practitioners —Nonlinear continuous-time systems, being ubiquitous in engineering practice, are widely employed due to their versatility and effectiveness. Aiming at such systems, a $\mathcal{Q}$ -learning approach with optimal feature is proposed to strengthen control efficiency while reduce costs. However, it is well known that accurately capturing all the dynamic information of the system is a formidable task in practical operation. This defect inevitably weakens the feasibility of model-based control algorithms. Since the $\mathcal{Q}$ -learning algorithm presented in this paper does not require any dynamic knowledge of systems, it is promising enabler in enhancing the effectiveness and flexibility of engineering activities.

Discrete-time optimal control scheme based on Q-learning algorithm

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Discrete-Time Deterministic $Q$ -Learning: A Novel Convergence Analysis

Stochastic Linear Quadratic Optimal Control for Model-Free Discrete-Time Systems Based on Q-learning Algorithm

Iterative Q-learning-based Nonlinear Optimal Tracking Control

Q-learning for Optimal Control of Continuous-time Systems

An Optimal Tracking Control Method with Q-learning for Discrete-time Linear Switched System

Nonlinear Neuro-Optimal Tracking Control via Stable Iterative Q-Learning Algorithm

A Q-Learning Algorithm for Discrete-Time Linear-Quadratic Control with Random Parameters of Unknown Distribution: Convergence and Stabilization

Q-Learning for Linear Quadratic Optimal Control with Terminal State Constraint

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Discrete-time adaptive iterative learning control with unknown control directions

Optimal Control for Continuous-Time Unknown Nonlinear Affine Systems: A $\Mathcal{q}$-Learning Approach

Event-triggered Suboptimal Control Based Adaptive Reinforcement Learning.

Reinforcement Learning with Adjustable Convergence Rate for Data-Based Nonlinear Control

Learning Algorithm for LQG Model with Constrained Control

Model-Free Optimal Tracking Design With Evolving Control Strategies via Q-Learning

Model-Free Optimal Tracking Control Via Critic-Only Q-Learning.

Discrete-Time Nonlinear Optimal Control Using Multi-Step Reinforcement Learning

Generalized Policy Iteration-based Reinforcement Learning Algorithm for Optimal Control of Unknown Discrete-time Systems

Adaptive Constrained Optimal Control Design for Data-Based Nonlinear Discrete-Time Systems With Critic-Only Structure