Abstract:The recursive least-squares (RLS) algorithm is one of the most well-known algorithms used in adaptive filtering, system identification and adaptive control. Its popularity is mainly due to its fast convergence speed, which is considered to be optimal in practice. In this paper, RLS methods are used to solve reinforcement learning problems, where two new reinforcement learning algorithms using linear value function approximators are proposed and analyzed. The two algorithms are called RLS-TD(lambda) and Fast-AHC (Fast Adaptive Heuristic Critic), respectively. RLS-TD(lambda) can be viewed as the extension of RLS-TD(0) from lambda=0 to general lambda within interval [0,1], so it is a multi-step temporal-difference (TD) learning algorithm using RLS methods. The convergence with probability one and the limit of convergence of RLS-TD(lambda) are proved for ergodic Markov chains. Compared to the existing LS-TD(lambda) algorithm, RLS-TD(lambda) has advantages in computation and is more suitable for online learning. The effectiveness of RLS-TD(lambda) is analyzed and verified by learning prediction experiments of Markov chains with a wide range of parameter settings. The Fast-AHC algorithm is derived by applying the proposed RLS-TD(lambda) algorithm in the critic network of the adaptive heuristic critic method. Unlike conventional AHC algorithm, Fast-AHC makes use of RLS methods to improve the learning-prediction efficiency in the critic. Learning control experiments of the cart-pole balancing and the acrobot swing-up problems are conducted to compare the data efficiency of Fast-AHC with conventional AHC. From the experimental results, it is shown that the data efficiency of learning control can also be improved by using RLS methods in the learning-prediction process of the critic. The performance of Fast-AHC is also compared with that of the AHC method using LS-TD(lambda). Furthermore, it is demonstrated in the experiments that different initial values of the variance matrix in RLS-TD(lambda) are required to get better performance not only in learning prediction but also in learning control. The experimental results are analyzed based on the existing theoretical work on the transient phase of forgetting factor RLS methods.

A Fast Recursive Total Least Square Algorithm for Finite Impulse Response Filters

Fast Recursive Least Squares Adaptive Filtering by Fast Fourier Transform-Based Conjugate Gradient Iterations

A recursive adaptive lattice filters for impulsive signals

Optimal Algorithm for FIR Digital Filter with Canonical Signed Digit Coefficients

A Fast Proportionate RLS Adaptive Equalization for Underwater Acoustic Communications

Adaptive Kalman Filtering Developed from Recursive Least Squares Forgetting Algorithms

Discrete Time $q$-Lag Maximum Likelihood FIR Smoothing and Iterative Recursive Algorithm

Synthesis of quasi-optimal fast filters by the least square criterion

Fast Iterative Reweighted Least Squares Algorithm for Sparse Signals Recovery.

Application of Fast Kalman Estimation to Adaptive Equalization

A Simple Algorithm for Constrained Least Square Design for FIR Filters

A Novel Adaptive Algorithm Using in UWB RAKE Receiver: Fast Transversal Filter Algorithm

Novel adaptive algorithm using in UWB RAKE receiver: Fast transversal filter algorithm

Recursive and non-recursive filters for sequential smoothing and prediction with instantaneous phase and frequency estimation applications (extended version)

Conjugate gradient acceleration of iteratively re-weighted least squares methods

Algorithm of Variable Transition Bandwidth FIR Filters with Weighted-Least-square Method

Fast Sampling for Linear Inverse Problems of Vectors and Tensors using Multilinear Extensions

Constructing numerically stable Kalman filter-based algorithms for gradient-based adaptive filtering

Recursive Total Least-Squares Algorithm Based on Inverse Power Method and Dichotomous Coordinate-Descent Iterations

Conservation of the passband signal amplitude using a filter based on the Fast Fourier Transform algorithm

Efficient Reinforcement Learning Using Recursive Least-Squares Methods