Abstract:As an important algorithm in deep reinforcement learning, advantage actor critic (A2C) has been widely succeeded in both discrete and continuous control tasks with raw pixel inputs, but its sample efficiency still needs to improve more. In traditional reinforcement learning, actor-critic algorithms generally use the recursive least squares (RLS) technology to update the parameter of linear function approximators for accelerating their convergence speed. However, A2C algorithms seldom use this technology to train deep neural networks (DNNs) for improving their sample efficiency. In this paper, we propose two novel RLS-based A2C algorithms and investigate their performance. Both proposed algorithms, called RLSSA2C and RLSNA2C, use the RLS method to train the critic network and the hidden layers of the actor network. The main difference between them is at the policy learning step. RLSSA2C uses an ordinary first-order gradient descent algorithm and the standard policy gradient to learn the policy parameter. RLSNA2C uses the Kronecker-factored approximation, the RLS method and the natural policy gradient to learn the compatible parameter and the policy parameter. In addition, we analyze the complexity and convergence of both algorithms, and present three tricks for further improving their convergence speed. Finally, we demonstrate the effectiveness of both algorithms on 40 games in the Atari 2600 environment and 11 tasks in the MuJoCo environment. From the experimental results, it is shown that our both algorithms have better sample efficiency than the vanilla A2C on most games or tasks, and have higher computational efficiency than other two state-of-the-art algorithms.

Efficient Reinforcement-Learning Control Algorithm Using Experience Reuse

Natural Gradient Based Reinforcement Learning Algorithm Using Active Stimulating

Model Reference Output Feedback Control Using Episodic Natural Actor-Critic

Learning Linear Parameter-Varying Control of Small-Scale Helicopter Using Episodic Natural Actor-Critic Method

A Novel Heterogeneous Actor-critic Algorithm with Recent Emphasizing Replay Memory

Episodic Reinforcement Learning with Expanded State-reward Space

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

Deep Deterministic Policy Gradient with Episode Experience Replay

Asynchronous Episodic Deep Deterministic Policy Gradient: Toward Continuous Control in Computationally Complex Environments

Experience Replay for Least-Squares Policy Iteration

An Active Exploration Method for Data Efficient Reinforcement Learning

Neural Episodic Control with State Abstraction

Episodic Reinforcement Learning with Associative Memory.

Policy Gradient Adaptive Critic Designs for Model-Free Optimal Tracking Control With Experience Replay

Efficient Data Use in Incremental Actor–critic Algorithms

Enhancing Off-Policy Constrained Reinforcement Learning Through Adaptive Ensemble C Estimation

Adaptive Evolutionary Reinforcement Learning with Policy Direction

Batch process control based on reinforcement learning with segmented prioritized experience replay

Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past

AutoEG: Automated Experience Grafting for Off-Policy Deep Reinforcement Learning

Recursive Least Squares Advantage Actor-Critic Algorithms