Abstract:Deep reinforcement learning based on the asynchronous method is a new kind of reinforcement learning. It takes a multithreading way to enable multiple agents to update the parameters asynchronously in different exploration spaces. In this way, agents no longer need experience to reply and can update parameters online. At the same time, the asynchronous method can greatly improve the convergence speed of the algorithms and significantly improve the convergence performance of the algorithms. Asynchronous deep reinforcement learning algorithms, especially asynchronous advantage actor-critic algorithm, are very effective in solving practical problems and have been widely used. However, in existing asynchronous deep reinforcement learning algorithms, when each thread pushes updates to the global thread, it adopts a uniform learning rate, and fails to take account of the different information transmitted by different threads at each update. When the update of the agent to global thread is more biased towards failure information, it has no obvious help to update the parameters of the learning system. Therefore, we introduce the dynamic weights to asynchronous deep reinforcement learning algorithms and propose a new reinforcement learning algorithm named asynchronous advantage actor-critic with dynamic updating weights (DWA3C). When the information pushed by an agent is obviously helpful for the improvement of the system performance, we will enhance the update range, otherwise, we will weaken that. In this way, we can significantly improve the convergence efficiencies and convergence performances of the asynchronous deep reinforcement learning algorithms. And we also test the effectiveness of the algorithm through experiments. The experimental results show that, in the same running time, the proposed algorithm can significantly improve the convergence efficiency and convergence performance compared with the existing algorithms.

Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup

Robustness Assessment of Asynchronous Advantage Actor-Critic Based on Dynamic Skewness and Sparseness Computation: A Parallel Computing View

Averaged-A3C for Asynchronous Deep Reinforcement Learning.

Recursive Least Squares Advantage Actor-Critic Algorithms

ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages

Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU

Efficient Parallel Methods for Deep Reinforcement Learning

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Finite-Time Analysis of Decentralized Single-Timescale Actor-Critic

Optimal Elevator Group Control via Deep Asynchronous Actor–Critic Learning

Applications of Asynchronous Deep Reinforcement Learning Based on Dynamic Updating Weights

Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation

Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis

Efficient Data Use in Incremental Actor–critic Algorithms

Actor-Critic Reinforcement Learning with Phased Actor

Asynchronous learning for actor-critic neural networks and synchronous triggering for multiplayer system

Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality

A Single-Loop Deep Actor-Critic Algorithm for Constrained Reinforcement Learning with Provable Convergence

ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization

Asynchronous Parallel Policy Gradient Methods for the Linear Quadratic Regulator

Heavy-Ball Momentum Accelerated Actor-Critic With Function Approximation