Abstract:In this paper, we present a kernel-based least squares policy iteration (KLSPI) algorithm for reinforcement learning (RL) in large or continuous state spaces, which can be used to realize adaptive feedback control of uncertain dynamic systems. By using KLSPI, near-optimal control policies can be obtained without much a priori knowledge on dynamic models of control plants. In KLSPI, Mercer kernels are used in the policy evaluation of a policy iteration process, where a new kernel-based least squares temporal-difference algorithm called KLSTD-Q is proposed for efficient policy evaluation. To keep the sparsity and improve the generalization ability of KLSTD-Q solutions, a kernel sparsification procedure based on approximate linear dependency (ALD) is performed. Compared to the previous works on approximate RL methods, KLSPI makes two progresses to eliminate the main difficulties of existing results. One is the better convergence and (near) optimality guarantee by using the KLSTD-Q algorithm for policy evaluation with high precision. The other is the automatic feature selection using the ALD-based kernel sparsification. Therefore, the KLSPI algorithm provides a general RL method with generalization performance and convergence guarantee for large-scale Markov decision problems (MDPs). Experimental results on a typical RL task for a stochastic chain problem demonstrate that KLSPI can consistently achieve better learning efficiency and policy quality than the previous least squares policy iteration (LSPI) algorithm. Furthermore, the KLSPI method was also evaluated on two nonlinear feedback control problems, including a ship heading control problem and the swing up control of a double-link underactuated pendulum called acrobot. Simulation results illustrate that the proposed method can optimize controller performance using little a priori information of uncertain dynamic systems. It is also demonstrated that KLSPI can be applied to online learning control by incorporating an initial controller to ensure online performance.

Nonparametric approximation policy iteration reinforcement learning based on Dyna framework

Nonparametric approximation generalized policy iteration reinforcement learning algorithm based on states clustering

Efficient Reinforcement Learning in Continuous State and Action Spaces with Dyna and Policy Approximation.

Nonparametric Approximation Policy Iteration Reinforcement Learning Based on CMAC

A Heuristic Dyna Optimizing Algorithm Using Approximate Model Representation

Dyna-Validator: A Model-based Reinforcement Learning Method with Validated Simulated Experiences.

Efficient Reinforcement-Learning Control Algorithm Using Experience Reuse

Approximate Policy Iteration With Deep Minimax Average Bellman Error Minimization

Adaptive Sample Collection Using Active Learning for Kernel-Based Approximate Policy Iteration

Data-based Approximate Policy Iteration for Affine Nonlinear Continuous-Time Optimal Control Design.

Hierarchical Approximate Policy Iteration with Binary-Tree State Space Decomposition.

Integrating symmetry of environment by designing special basis functions for value function approximation in reinforcement learning

A Novel Policy Iteration Algorithm for Nonlinear Continuous-Time H$\infty$ Control Problem

An Active Exploration Method for Data Efficient Reinforcement Learning

A Phased Dyna Reinforcement Learning Algorithm

Approximate Policy-Based Accelerated Deep Reinforcement Learning.

An Optimized Dyna Architecture Algorithm with Prioritized Sweeping

Experience Replay for Least-Squares Policy Iteration

Kernel-Based Least Squares Policy Iteration for Reinforcement Learning.

Online Adaptive Policy Learning Algorithm for H-Infinity State Feedback Control of Unknown Affine Nonlinear Discrete-Time Systems

Design of Nonlinear Iterative Learning Control Based on Deep Reinforcement Learning Algorithm