Abstract:This paper revisits the estimation bias control problem of Q-learning, motivated by the fact that the estimation bias is not always evil, i.e., some environments benefit from overestimation bias or underestimation bias, while others suffer from these biases. Different from previous coarse-grained bias control methods, this paper proposes a fine-grained bias control algorithm called Order Q-learning. It uses the order statistic of multiple independent Q-tables to control bias and flexibly meet the personalized bias needs of different environments, i.e., the bias can vary from underestimation bias to overestimation bias as one selects a higher order Q-value. We derive the expected estimation bias and its lower bound and upper bound. They reveal that the expected estimation bias is inversely proportional to the number of Q-tables and proportional to the index of order statistic function. To show the versatility of Order Q-learning, we design an adaptive parameter adjustment strategy, leading to AdaOrder (Adaptive Order) Q-learning. It adaptively selects the number of Q-tables and the index of order statistic function via the number of visits to state-action pair and the average Q-value. We extend Order Q-learning and AdaOrder Q-learning to the large scale setting with function approximation, leading to Order DQN and AdaOrder DQN, respectively. Finally, we consider two experiment settings: deep reinforcement learning experiments show that our method outperforms several SOTA baselines drastically; tabular MDP experiments reveal fundamental insights into why our method can achieve superior performance.Our supplementary file can be found in https://1drv.ms/f/s!Atddp1iaDmL2gjv31CaGquw5WwYI.

Self-correcting Q-learning.

Deep Reinforcement Learning with Double Q-Learning

Successively Pruned Q-Learning: Using Self Q-function to Reduce the Overestimation.

On the Estimation Bias in Double Q-Learning

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

Deep Q Net Based on Advantage Learning

Finite-Time Analysis of Simultaneous Double Q-learning

Stochastic Double Deep Q-Network.

Double Speedy Q-Learning Based on Successive over Relaxation

Ensemble Bootstrapping for Q-Learning

Accurate Q-Learning.

Decorrelated Double Q-learning

Suppressing Overestimation in Q-Learning through Adversarial Behaviors

Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods

Adapting Double Q-Learning for Continuous Reinforcement Learning

Adaptive Moving Average Q-learning

A Dynamic Adjusting Reward Function Method for Deep Reinforcement Learning with Adjustable Parameters

Adaptive Order Q-learning

Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning

Integrated Double Estimator Architecture for Reinforcement Learning

Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples