Abstract:The recursive least-squares (RLS) algorithm is one of the most well-known algorithms used in adaptive filtering, system identification and adaptive control. Its popularity is mainly due to its fast convergence speed, which is considered to be optimal in practice. In this paper, RLS methods are used to solve reinforcement learning problems, where two new reinforcement learning algorithms using linear value function approximators are proposed and analyzed. The two algorithms are called RLS-TD(lambda) and Fast-AHC (Fast Adaptive Heuristic Critic), respectively. RLS-TD(lambda) can be viewed as the extension of RLS-TD(0) from lambda=0 to general lambda within interval [0,1], so it is a multi-step temporal-difference (TD) learning algorithm using RLS methods. The convergence with probability one and the limit of convergence of RLS-TD(lambda) are proved for ergodic Markov chains. Compared to the existing LS-TD(lambda) algorithm, RLS-TD(lambda) has advantages in computation and is more suitable for online learning. The effectiveness of RLS-TD(lambda) is analyzed and verified by learning prediction experiments of Markov chains with a wide range of parameter settings. The Fast-AHC algorithm is derived by applying the proposed RLS-TD(lambda) algorithm in the critic network of the adaptive heuristic critic method. Unlike conventional AHC algorithm, Fast-AHC makes use of RLS methods to improve the learning-prediction efficiency in the critic. Learning control experiments of the cart-pole balancing and the acrobot swing-up problems are conducted to compare the data efficiency of Fast-AHC with conventional AHC. From the experimental results, it is shown that the data efficiency of learning control can also be improved by using RLS methods in the learning-prediction process of the critic. The performance of Fast-AHC is also compared with that of the AHC method using LS-TD(lambda). Furthermore, it is demonstrated in the experiments that different initial values of the variance matrix in RLS-TD(lambda) are required to get better performance not only in learning prediction but also in learning control. The experimental results are analyzed based on the existing theoretical work on the transient phase of forgetting factor RLS methods.

Application of Adaptive Forgetting Factor RLS Algorithm in Target Tracking

Low-complexity Correlated Time-Averaged Variable Forgetting Factor Mechanism for Diffusion RLS Algorithm in Sensor Networks

RASTMTrack: Robust and Adaptive Space-Time Memory Networks for Visual Tracking

Infrared Point Target Detection Based on Exponentially Weighted RLS Algorithm and Dual Solution Improvement

An Improved Hybrid Beamforming Algorithm for Fast Target Tracking in Satellite and V2X Communication

Adaptive Coordinated Motion Control With Variable Forgetting Factor For A Dual-Arm Space Robot In Post-Capture Of A Noncooperative Target

Adaptive And Compressive Target Tracking Based On Feature Point Matching

A long-memory pedestrian target tracking algorithm incorporating spatiotemporal trajectory feature enhancement model

Variable Forgetting Factor Mechanisms for Diffusion Recursive Least Squares Algorithm in Sensor Networks.

Bearing-Only Target Tracking Using Cubature Rauch-Tung-Striebel Smoother

Adaptive Correlation Filters with Long-Term and Short-Term Memory for Object Tracking

A Perceptually Adaptive Long-Term Tracking Method for the Complete Occlusion and Disappearance of a Target

A fuzzy adaptive algorithm based on current statistical model for maneuvering target tracking

An Adaptive Tracking Method for Moving Target in Fluctuating Reverberation Environment

Low-complexity Variable Forgetting Factor Mechanism for Blind Adaptive Constrained Constant Modulus Algorithms

FACT: Feature Adaptive Continual-learning Tracker for Multiple Object Tracking

Efficient Reinforcement Learning Using Recursive Least-Squares Methods

Selective Memory Recursive Least Squares: Recast Forgetting into Memory in RBF Neural Network Based Real-Time Learning

Adaptive visual tracking with failure detection

An Improved Kalman Filter Based on Long Short-Memory Recurrent Neural Network for Nonlinear Radar Target Tracking

Adaptive Tracking of High-Maneuvering Targets Based on Multi-Feature Fusion Trajectory Clustering: LPI's Purpose