Abstract:Inspired by Double Q-learning algorithm, the Double-DQN (DDQN) algorithm was originally proposed in order to address the overestimation issue in the original DQN algorithm. The DDQN has successfully shown both theoretically and empirically the importance of decoupling in terms of action evaluation and selection in computation of target values; although, all the benefits were acquired with only a simple adaption to DQN algorithm, minimal possible change as it was mentioned by the authors. Nevertheless, there seems a roll-back in the proposed algorithm of DDQN since the parameters of policy network are emerged again in the target value function which were initially withdrawn by DQN with the hope of tackling the serious issue of moving targets and the instability caused by it (i.e., by moving targets) in the process of learning. Therefore, in this paper three modifications to the DDQN algorithm are proposed with the hope of maintaining the performance in the terms of both stability and overestimation. These modifications are focused on the logic of decoupling the best action selection and evaluation in the target value function and the logic of tackling the moving targets issue. Each of these modifications have their own pros and cons compared to the others. The mentioned pros and cons mainly refer to the execution time required for the corresponding algorithm and the stability provided by the corresponding algorithm. Also, in terms of overestimation, none of the modifications seem to underperform compared to the original DDQN if not outperform it. With the intention of evaluating the efficacy of the proposed modifications, multiple empirical experiments along with theoretical experiments were conducted. The results obtained are represented and discussed in this article.

Successively Pruned Q-Learning: Using Self Q-function to Reduce the Overestimation.

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

Self-correcting Q-learning.

Gradient Q(σ, Λ): A Unified Algorithm with Function Approximation for Reinforcement Learning

On the Estimation Bias in Double Q-Learning

Maxmin Q-learning: Controlling the Estimation Bias of Q-learning

Deep Reinforcement Learning with Double Q-Learning

Adaptive Order Q-learning

Suppressing Overestimation in Q-Learning through Adversarial Behaviors

A controlling estimation bias method: Max_Mix_Min estimator for Q-learning

Randomized Ensembled Double Q-Learning: Learning Fast Without a Model

Adapting Double Q-Learning for Continuous Reinforcement Learning

An Overestimation Reduction Method Based on the Multi-step Weighted Double Estimation Using Value-Decomposition Multi-agent Reinforcement Learning

A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation

Modified Double DQN: addressing stability

Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods

Finite-Time Analysis of Simultaneous Double Q-learning

Ensemble Bootstrapping for Q-Learning

Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback

Action Candidate Based Clipped Double Q-learning for Discrete and Continuous Action Tasks

Controlling Estimation Error in Reinforcement Learning via Reinforced Operation