Abstract:The recursive least-squares (RLS) algorithm is one of the most well-known algorithms used in adaptive filtering, system identification and adaptive control. Its popularity is mainly due to its fast convergence speed, which is considered to be optimal in practice. In this paper, RLS methods are used to solve reinforcement learning problems, where two new reinforcement learning algorithms using linear value function approximators are proposed and analyzed. The two algorithms are called RLS-TD(lambda) and Fast-AHC (Fast Adaptive Heuristic Critic), respectively. RLS-TD(lambda) can be viewed as the extension of RLS-TD(0) from lambda=0 to general lambda within interval [0,1], so it is a multi-step temporal-difference (TD) learning algorithm using RLS methods. The convergence with probability one and the limit of convergence of RLS-TD(lambda) are proved for ergodic Markov chains. Compared to the existing LS-TD(lambda) algorithm, RLS-TD(lambda) has advantages in computation and is more suitable for online learning. The effectiveness of RLS-TD(lambda) is analyzed and verified by learning prediction experiments of Markov chains with a wide range of parameter settings. The Fast-AHC algorithm is derived by applying the proposed RLS-TD(lambda) algorithm in the critic network of the adaptive heuristic critic method. Unlike conventional AHC algorithm, Fast-AHC makes use of RLS methods to improve the learning-prediction efficiency in the critic. Learning control experiments of the cart-pole balancing and the acrobot swing-up problems are conducted to compare the data efficiency of Fast-AHC with conventional AHC. From the experimental results, it is shown that the data efficiency of learning control can also be improved by using RLS methods in the learning-prediction process of the critic. The performance of Fast-AHC is also compared with that of the AHC method using LS-TD(lambda). Furthermore, it is demonstrated in the experiments that different initial values of the variance matrix in RLS-TD(lambda) are required to get better performance not only in learning prediction but also in learning control. The experimental results are analyzed based on the existing theoretical work on the transient phase of forgetting factor RLS methods.

RKRLS and Its Application to Modeling and Prediction of Rubber Compound Quality

Recursive Least Squares Support Vector Regression

DISCOUNTED-MEASUREMENT RPLS ALGORITHM AND ITS APPLICATION TO QUALITY CONTROL OF RUBBER MIXING PROCESS

Online dual updating with recursive PLS model and its application in predicting crystal size of purified terephthalic acid (PTA) process

Quality Monitoring of the Rubber Mixing Process by Using the Discounted-Measurement RPLS Algorithm.

Robust Recursive Kernel Learning Modeling Method with Application to Blast Furnace

Adaptive recursive kernel learning with application to online quality prediction of industrial rubber mixing process

RPLS Based Adaptive Statistical Quality Monitoring of Rubber Mixing Process

Modeling and Prediction Based on Recursive Least Square Support Vector Regression

Adaptive least contribution elimination kernel learning approach for rubber mixing soft-sensing modeling

Semi-Supervised Hybrid Local Kernel Regression for Soft Sensor Modelling of Rubber-Mixing Process

Recursive Least-Squares Method with Membership Functions

Online prediction of Mooney viscosity in industrial rubber mixing process via adaptive kernel learning method

Modified kernel Hebbian algorithm with application to modeling of hydro-dearomatization process

Nonlinear Recursive Partial Least Squares and Its Applications

GMPLS Based Intelligent Quality Control for Internal Rubber Mixing Process

Efficient Reinforcement Learning Using Recursive Least-Squares Methods

Robust semi-supervised modelling method and its application to fault detection in chemical processes

Selective Recursive LSSVR and its Applications in Process Modeling

Adaptive Local Kernel-Based Learning for Soft Sensor Modeling of Nonlinear Processes

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces