Abstract:In this paper, we present a kernel-based least squares policy iteration (KLSPI) algorithm for reinforcement learning (RL) in large or continuous state spaces, which can be used to realize adaptive feedback control of uncertain dynamic systems. By using KLSPI, near-optimal control policies can be obtained without much a priori knowledge on dynamic models of control plants. In KLSPI, Mercer kernels are used in the policy evaluation of a policy iteration process, where a new kernel-based least squares temporal-difference algorithm called KLSTD-Q is proposed for efficient policy evaluation. To keep the sparsity and improve the generalization ability of KLSTD-Q solutions, a kernel sparsification procedure based on approximate linear dependency (ALD) is performed. Compared to the previous works on approximate RL methods, KLSPI makes two progresses to eliminate the main difficulties of existing results. One is the better convergence and (near) optimality guarantee by using the KLSTD-Q algorithm for policy evaluation with high precision. The other is the automatic feature selection using the ALD-based kernel sparsification. Therefore, the KLSPI algorithm provides a general RL method with generalization performance and convergence guarantee for large-scale Markov decision problems (MDPs). Experimental results on a typical RL task for a stochastic chain problem demonstrate that KLSPI can consistently achieve better learning efficiency and policy quality than the previous least squares policy iteration (LSPI) algorithm. Furthermore, the KLSPI method was also evaluated on two nonlinear feedback control problems, including a ship heading control problem and the swing up control of a double-link underactuated pendulum called acrobot. Simulation results illustrate that the proposed method can optimize controller performance using little a priori information of uncertain dynamic systems. It is also demonstrated that KLSPI can be applied to online learning control by incorporating an initial controller to ensure online performance.

Reinforcement Learning for Linear Exponential Quadratic Gaussian Problem

Average Cost Optimal Control of Stochastic Systems Using Reinforcement Learning

Robust Reinforcement Learning for Risk-Sensitive Linear Quadratic Gaussian Control

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

A Q-Learning Algorithm for Discrete-Time Linear-Quadratic Control with Random Parameters of Unknown Distribution: Convergence and Stabilization

Reinforcement Learning for a Discrete-Time Linear-Quadratic Control Problem with an Application

Data-driven policy iteration algorithm for continuous-time stochastic linear-quadratic optimal control problems

Value iteration for LQR control of unknown stochastic-parameter linear systems

Model-free Optimal Control of Discrete-Time Systems with Additive and Multiplicative Noises

Fast Policy Learning for Linear Quadratic Control with Entropy Regularization

Inverse Reinforcement Learning for Discrete-Time Linear Quadratic Systems

Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problems

Policy Iteration Based Q-learning for Linear Nonzero-Sum Quadratic Differential Games.

Model-free $H_{\infty}$ control of Itô stochastic system via off-policy reinforcement learning

Policy gradient methods for discrete time linear quadratic regulator with random parameters

Off-Policy Q-Learning for Infinite Horizon LQR Problem with Unknown Dynamics

Risk Sensitive, Nonlinear Optimal Control: Iterative Linear Exponential-Quadratic Optimal Control with Gaussian Noise

Model-free Reinforcement Learning for H_2/H_∞ Control of Stochastic Discrete-time Systems

Kernel-Based Least Squares Policy Iteration for Reinforcement Learning.

Sublinear Regret for a Class of Continuous-Time Linear--Quadratic Reinforcement Learning Problems

Stochastic LQ optimal control for Markov jumping systems with multiplicative noise using reinforcement learning