Abstract:The recursive least-squares (RLS) algorithm is one of the most well-known algorithms used in adaptive filtering, system identification and adaptive control. Its popularity is mainly due to its fast convergence speed, which is considered to be optimal in practice. In this paper, RLS methods are used to solve reinforcement learning problems, where two new reinforcement learning algorithms using linear value function approximators are proposed and analyzed. The two algorithms are called RLS-TD(lambda) and Fast-AHC (Fast Adaptive Heuristic Critic), respectively. RLS-TD(lambda) can be viewed as the extension of RLS-TD(0) from lambda=0 to general lambda within interval [0,1], so it is a multi-step temporal-difference (TD) learning algorithm using RLS methods. The convergence with probability one and the limit of convergence of RLS-TD(lambda) are proved for ergodic Markov chains. Compared to the existing LS-TD(lambda) algorithm, RLS-TD(lambda) has advantages in computation and is more suitable for online learning. The effectiveness of RLS-TD(lambda) is analyzed and verified by learning prediction experiments of Markov chains with a wide range of parameter settings. The Fast-AHC algorithm is derived by applying the proposed RLS-TD(lambda) algorithm in the critic network of the adaptive heuristic critic method. Unlike conventional AHC algorithm, Fast-AHC makes use of RLS methods to improve the learning-prediction efficiency in the critic. Learning control experiments of the cart-pole balancing and the acrobot swing-up problems are conducted to compare the data efficiency of Fast-AHC with conventional AHC. From the experimental results, it is shown that the data efficiency of learning control can also be improved by using RLS methods in the learning-prediction process of the critic. The performance of Fast-AHC is also compared with that of the AHC method using LS-TD(lambda). Furthermore, it is demonstrated in the experiments that different initial values of the variance matrix in RLS-TD(lambda) are required to get better performance not only in learning prediction but also in learning control. The experimental results are analyzed based on the existing theoretical work on the transient phase of forgetting factor RLS methods.

A Learning Algorithm of CMAC Based on RLS

Parameter Self-Tuning of SISO Compact-Form Model-Free Adaptive Controller Based on Long Short-Term Memory Neural Network

Multi-loop Adaptive Internal Model Control Based on a Dynamic Partial Least Squares Model

An Engineering-Effective Adaptive Model Algorithm Control (AMAC) and Simulation

The Imc Structure of Multi-Rate Multivariable Predictive Control Systems and an Improved Algorithm

CMAC-Based Learning Control for a Batch Reactor

AN CMAC NETWORK MODEL AND ITS APPLICATION TO ROBOTIC CONTROL

ELMCS Design Using CMAC Compound Control Algorithm

Continuous CMAC-QRLS and Its Systolic Array

Efficient Reinforcement Learning Using Recursive Least-Squares Methods

A Robust Adaptive CMAC Neural Network-Based Multisliding Mode Control Method for Unmatched Uncertain Nonlinear Systems

Fuzzy CMAC with Online Learning Ability and Its Application

CMAC-PID Compound Controller Research for Water Loading System

Random Re-Connection Leaning Algorithm of CMAC Model in Prosthetic Knee Control

Accelerated reinforcement learning control using modified CMAC neural networks

Selective Partial-Update Augmented Complex-Valued LMS Algorithm and Its Performance Analysis.

Design and Simulation of Adaptive CMAC Controller

Hybrid Position/force Controller of Robot Manipulators Based on CMAC Learning

Design of Stable Control Systems Based on Reinforcement Fuzzy CMAC

CAAC: an Effective Reinforcement Learning Algorithm for Sparse Reward in Automatic Control Systems

Arbitrarily Fast Tracking Multivariable Least-squares MRAC