Abstract:The recursive least-squares (RLS) algorithm is one of the most well-known algorithms used in adaptive filtering, system identification and adaptive control. Its popularity is mainly due to its fast convergence speed, which is considered to be optimal in practice. In this paper, RLS methods are used to solve reinforcement learning problems, where two new reinforcement learning algorithms using linear value function approximators are proposed and analyzed. The two algorithms are called RLS-TD(lambda) and Fast-AHC (Fast Adaptive Heuristic Critic), respectively. RLS-TD(lambda) can be viewed as the extension of RLS-TD(0) from lambda=0 to general lambda within interval [0,1], so it is a multi-step temporal-difference (TD) learning algorithm using RLS methods. The convergence with probability one and the limit of convergence of RLS-TD(lambda) are proved for ergodic Markov chains. Compared to the existing LS-TD(lambda) algorithm, RLS-TD(lambda) has advantages in computation and is more suitable for online learning. The effectiveness of RLS-TD(lambda) is analyzed and verified by learning prediction experiments of Markov chains with a wide range of parameter settings. The Fast-AHC algorithm is derived by applying the proposed RLS-TD(lambda) algorithm in the critic network of the adaptive heuristic critic method. Unlike conventional AHC algorithm, Fast-AHC makes use of RLS methods to improve the learning-prediction efficiency in the critic. Learning control experiments of the cart-pole balancing and the acrobot swing-up problems are conducted to compare the data efficiency of Fast-AHC with conventional AHC. From the experimental results, it is shown that the data efficiency of learning control can also be improved by using RLS methods in the learning-prediction process of the critic. The performance of Fast-AHC is also compared with that of the AHC method using LS-TD(lambda). Furthermore, it is demonstrated in the experiments that different initial values of the variance matrix in RLS-TD(lambda) are required to get better performance not only in learning prediction but also in learning control. The experimental results are analyzed based on the existing theoretical work on the transient phase of forgetting factor RLS methods.

A Fractional Filter Based on Reinforcement Learning for Effective Tracking Under Impulsive Noise

A recursive adaptive lattice filters for impulsive signals

A New Virtual Tracking Sub-Algorithm Based Hybrid Active Control System for Narrowband Noise with Impulsive Interference

Fractional-Order Control Method Based on Twin-Delayed Deep Deterministic Policy Gradient Algorithm

Imm Algorithm Based On H-Infinity Filter For Maneuvering Target Tracking

Projection FxLMS Framework of Active Noise Control Against Impulsive Noise Environments

An Innovative Parameter Estimation for Fractional Order Systems with Impulse Noise.

IMM Algorithm Based on H∞ Filter for Maneuvering Target Tracking

Reinforcement Learning for Input Constrained Sub-optimal Tracking Control in Discrete-time Two-time-scale Systems

Fractional lower order moment based adaptive algorithms for active noise control of impulsive noise sources

Command Filter-Based Adaptive Fuzzy Finite-Time Tracking Control for Uncertain Fractional-Order Nonlinear Systems

An Extension to the Filtered-X LMS Algorithm with Logarithmic Transformation.

A Novel Fractional Nonlinear State Estimation Algorithm in Non-Gaussian Noise Environment

Unsupervised Robust Recursive Least-Squares Algorithm for Impulsive Noise Filtering

A Novel Adaptive State Detector-Based Post-Filtering Active Control Algorithm for Gaussian Noise Environment with Impulsive Interference

Efficient Reinforcement Learning Using Recursive Least-Squares Methods

An improved fuzzy inference strategy using reinforcement learning for trajectory-tracking of a mobile robot under a varying slip ratio

Noise Detection and Image Denoising Based on Fractional Calculus

Process Noise Identification Based Particle Filter: an Efficient Method to Track Highly Manoeuvring Targets

Robust multi-reference adaptive gain FxLMS algorithm for active impulsive noise control

Dynamical Hyperparameter Optimization Via Deep Reinforcement Learning in Tracking