Abstract:Inspired by Double Q-learning algorithm, the Double-DQN (DDQN) algorithm was originally proposed in order to address the overestimation issue in the original DQN algorithm. The DDQN has successfully shown both theoretically and empirically the importance of decoupling in terms of action evaluation and selection in computation of target values; although, all the benefits were acquired with only a simple adaption to DQN algorithm, minimal possible change as it was mentioned by the authors. Nevertheless, there seems a roll-back in the proposed algorithm of DDQN since the parameters of policy network are emerged again in the target value function which were initially withdrawn by DQN with the hope of tackling the serious issue of moving targets and the instability caused by it (i.e., by moving targets) in the process of learning. Therefore, in this paper three modifications to the DDQN algorithm are proposed with the hope of maintaining the performance in the terms of both stability and overestimation. These modifications are focused on the logic of decoupling the best action selection and evaluation in the target value function and the logic of tackling the moving targets issue. Each of these modifications have their own pros and cons compared to the others. The mentioned pros and cons mainly refer to the execution time required for the corresponding algorithm and the stability provided by the corresponding algorithm. Also, in terms of overestimation, none of the modifications seem to underperform compared to the original DDQN if not outperform it. With the intention of evaluating the efficacy of the proposed modifications, multiple empirical experiments along with theoretical experiments were conducted. The results obtained are represented and discussed in this article.

On the Reduction of Variance and Overestimation of Deep Q-Learning

Stochastic Variance Reduction for Deep Q-learning

Stochastic Modified Equations and Dynamics of Dropout Algorithm

Deep Q-Learning: Theoretical Insights from an Asymptotic Analysis

Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methods

Suppressing Overestimation in Q-Learning through Adversarial Behaviors

Variance aware reward smoothing for deep reinforcement learning

Deep Reinforcement Learning with Double Q-Learning

Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning

Sample Complexity of Variance-reduced Distributionally Robust Q-learning

Lifting the Veil: Unlocking the Power of Depth in Q-learning

Convergent and Efficient Deep Q Network Algorithm

An Information-Theoretic Optimality Principle for Deep Reinforcement Learning

Deep Q-Network for Stochastic Process Environments

On the Estimation Bias in Double Q-Learning

Controlling Estimation Error in Reinforcement Learning via Reinforced Operation

Modified Double DQN: addressing stability

Roll-Drop: accounting for observation noise with a single parameter

Dropout, a basic and effective regularization method for a deep learning model: a case study

Interpretable Option Discovery using Deep Q-Learning and Variational Autoencoders

Ensemble Bootstrapping for Q-Learning