Abstract:In this paper, we present a kernel-based least squares policy iteration (KLSPI) algorithm for reinforcement learning (RL) in large or continuous state spaces, which can be used to realize adaptive feedback control of uncertain dynamic systems. By using KLSPI, near-optimal control policies can be obtained without much a priori knowledge on dynamic models of control plants. In KLSPI, Mercer kernels are used in the policy evaluation of a policy iteration process, where a new kernel-based least squares temporal-difference algorithm called KLSTD-Q is proposed for efficient policy evaluation. To keep the sparsity and improve the generalization ability of KLSTD-Q solutions, a kernel sparsification procedure based on approximate linear dependency (ALD) is performed. Compared to the previous works on approximate RL methods, KLSPI makes two progresses to eliminate the main difficulties of existing results. One is the better convergence and (near) optimality guarantee by using the KLSTD-Q algorithm for policy evaluation with high precision. The other is the automatic feature selection using the ALD-based kernel sparsification. Therefore, the KLSPI algorithm provides a general RL method with generalization performance and convergence guarantee for large-scale Markov decision problems (MDPs). Experimental results on a typical RL task for a stochastic chain problem demonstrate that KLSPI can consistently achieve better learning efficiency and policy quality than the previous least squares policy iteration (LSPI) algorithm. Furthermore, the KLSPI method was also evaluated on two nonlinear feedback control problems, including a ship heading control problem and the swing up control of a double-link underactuated pendulum called acrobot. Simulation results illustrate that the proposed method can optimize controller performance using little a priori information of uncertain dynamic systems. It is also demonstrated that KLSPI can be applied to online learning control by incorporating an initial controller to ensure online performance.

Kernel dynamic policy programming: Applicable reinforcement learning to robot systems with high dimensional states

Ensemble Bootstrapped Deep Deterministic Policy Gradient For Vision-Based Robotic Grasping

Deep Model-Based Reinforcement Learning for Predictive Control of Robotic Systems with Dense and Sparse Rewards

Intelligent Control Of A Sensor-Actuator System Via Kernelized Least-Squares Policy Iteration

Pneumatic artificial muscle-driven robot control using local update reinforcement learning

Local Update Dynamic Policy Programming in reinforcement learning of pneumatic artificial muscle-driven humanoid hand control

Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models

Kernel-Based Decentralized Policy Evaluation for Reinforcement Learning

Reinforcement Learning Tracking Control for Robotic Manipulator With Kernel-Based Dynamic Model

Kernel-Based Least Squares Policy Iteration for Reinforcement Learning.

DOP: Deep Optimistic Planning with Approximate Value Function Evaluation

A kernel-based approximate dynamic programming approach: Theory and application

Dynamic Policy Programming with Descending Regularization for Efficient Reinforcement Learning Control

Reinforcement Learning in Continuous Time and Space

Improved Exploration through Latent Trajectory Optimization in Deep Deterministic Policy Gradient

Human-robot Force Cooperation Analysis by Deep Reinforcement Learning

A Modified Convergence DDPG Algorithm for Robotic Manipulation

Kalman meets Bellman: Improving Policy Evaluation through Value Tracking

Online Sparse Kernel Learning-Based Adaptive Dynamic Programming

Dynamic mean field programming