Abstract:This paper focuses on the linear-quadratic optimal control problem for unknown stochastic-parameter linear systems using reinforcement learning methods. Based on the second moments of random system matrices, a model-based value iteration algorithm is proposed to solve the problem and is proved to be convergent by using the contraction mapping theorem. For the case without knowing any information about the random system matrices, a normalized model-free value iteration algorithm is presented to learn the optimal control law by estimating the data for the next time. The collected data are normalized first in our model-free algorithm to reduce the error of the least squares method. It is proved that our algorithm can obtain an approximate optimal solution. Our algorithm is applicable for any distribution of the random system matrices and does not require an initial mean-square stabilizing control policy. Finally, an example illustrates that our algorithm converges to an approximate optimal control policy and that the normalization step can significantly reduce convergence errors.

Value Iteration Algorithm for Continuous-Time Linear Quadratic Stochastic Optimal Control Problems

Model-free Value Iteration Algorithm for Continuous-time Stochastic Linear Quadratic Optimal Control Problems

A policy iteration algorithm for non-Markovian control problems

Data-driven policy iteration algorithm for continuous-time stochastic linear-quadratic optimal control problems

Stochastic linear quadratic optimal control for continuous-time systems based on policy iteration

Value-Gradient Iteration with Quadratic Approximate Value Functions

Value iteration for LQR control of unknown stochastic-parameter linear systems

Convergence and Robustness of Value and Policy Iteration for the Linear Quadratic Regulator

Data-driven Policy Iteration Algorithm for Optimal Control of Continuous-Time Itô Stochastic Systems with Markovian Jumps

Robust policy iteration for continuous-time stochastic $H_\infty$ control problem with unknown dynamics

Policy Iteration for Multiplicative Noise Output Feedback Control

Linear quadratic optimal control problems for stochastic evolution equations in infinite horizon

Optimal State Tracking Control for Linear Discrete-time Systems Via Value Iteration

Policy Iteration Based Feedback Control

Finite Convergence of Value Iteration Algorithm for Discounted Infinite Horizon Optimal Control of Stochastic Logical Systems

Two Iterative Algorithms for Stochastic Algebraic Riccati Matrix Equations.

Optimal control for discrete-time affine non-linear systems using general value iteration

On-policy and Off-policy Value Iteration Algorithms for Stochastic Zero-Sum Games

Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problems

Approximate Finite-Horizon Optimal Control with Policy Iteration

Continuous-Time Distributed Policy Iteration for Multicontroller Nonlinear Systems