Abstract:This paper focuses on solving the linear quadratic regulator problem for discrete-time linear systems without knowing system matrices. The classical Q-learning methods for linear systems can be divided into Q-learning value iteration and Q-learning policy iteration. Q-learning value iteration converges at a linear convergence rate. Q-learning policy iteration has a second-order convergence rate but requires an initial stabilizing control policy. This paper aims to propose efficient model-free algorithms for solving the optimal control problem without requiring an initial stabilizing control policy. In this paper, we first present an equivalent problem for an auxiliary system with the same optimal control policy as the LQR problem A Q-learning algorithm is proposed to solve the equivalent problem, which is proven to converge monotonically to the optimal solution. The convergence rate of the Q-learning algorithm is heavily dependent on the auxiliary system, so we introduce a model-free homotopy method based on Q-learning to solve the LQR problem. This homotopy method can achieve the optimal solution in a finite number of iterations by solving an LQR problem in each iteration. Additionally, we propose a Q-learning Lyapunov iteration algorithm to solve the equivalent problem for an auxiliary system and analyze its properties. Finally, two examples are provided to demonstrate our results. Note to Practitioners —This paper proposes several Q-learning methods to solve the linear quadratic regulator problem for discrete-time linear systems. On the one hand, it is difficult to know the exact system dynamics knowledge in actual engineering, so this paper is devoted to developing model-free algorithms. On the other hand, this paper focuses on the LQR problem because it is widely spread in practical applications. We propose several model-free algorithms to solve the LQR problem, which provides the basis for optimal control of actual applications. Similar to policy iteration, our algorithms need to solve the Lyapunov equation. The advantage of our methods is that all of our algorithms do not have strict constraints on initial conditions compared with policy iteration. The properties of every algorithm proposed in this paper are provided. In addition, we focus on the efficiency of algorithms to obtain the optimal control policy faster. Two practical examples are used to verify the effectiveness of our methods. Finally, the applicable situations of each algorithm are summarized in the conclusion.

Numerical methods for closed-loop systems with non-autonomous data

Iterative design of suboptimal feedback control for bilinear parabolic PDE systems

The Quadratic-Quadratic Regulator Problem: Approximating feedback controls for quadratic-in-state nonlinear systems

Robust Adaptive Iterative Learning Control for Discrete-Time Nonlinear Systems With Time-Iteration-Varying Parameters.

Q-Learning Methods for LQR Control of Completely Unknown Discrete-Time Linear Systems

Learning to Control under Uncertainty with Data-Based Iterative Linear Quadratic Regulator

Exponentially Stable Adaptive Optimal Control of Uncertain LTI Systems

Numerical Schemes for Nonlinear Predictor Feedback

FRIDAY: Real-time Learning DNN-based Stable LQR controller for Nonlinear Systems under Uncertain Disturbances

Parallel and Proximal Constrained Linear-Quadratic Methods for Real-Time Nonlinear MPC

Numerical Discrete-Time Implementation of Continuous-Time Linear-Quadratic Model Predictive Control

Neural network optimal feedback control with enhanced closed loop stability

Dual-loop iterative optimal control for the finite horizon LQR problem with unknown dynamics

i2LQR: Iterative LQR for Iterative Tasks in Dynamic Environments

A Control-Theoretic Study on Runge-Kutta Methods with Application to Real-Time Fault-Tolerant Control of Nonlinear Systems

Adaptive Optimal Control with Guaranteed Convergence Rate for Continuous-Time Linear Systems with Completely Unknown Dynamics.

Numerical Methods for Optimal Control Problems with SPDEs

An Information-state based Approach to the Optimal Output Feedback Control of Nonlinear Systems

Numerical Discretization Methods for Linear Quadratic Control Problems with Time Delays

Comparative Study of Linear Control Strategies for Regulation, Tracking of Unstable Systems and the Scope for LQR Infusion in Optimisation Problems

Sequential Quadratic Programming-based Iterative Learning Control for Nonlinear Systems