Abstract:Deep reinforcement learning (DRL) methods has been widely applied on more and more challenging learning tasks, and achieved excellent performance. However, the efficiency of deep reinforcement learning is notoriously sensitive to their own hyperparameter configuration. The optimization process of deep reinforcement learning is highly dynamic and non-stationary, rather than a simple fitting process. So, its optimal hyperparameter should be adaptively adjusted according to the current learning process, rather than using a fixed set of hyperparameter configurations from beginning to end. DeepMind innovatively proposed a population based training (PBT) method for deep reinforcement learning, which achieved hyperparameter adaptation and made the model better trained. However, we assume that at the early stage when the learning model has little knowledge of the environment, frequent hyperparameter change will not be helpful for the model to learn efficiently, while learning with a reasonable fixed hyperparameter configuration will help the model obtain necessary knowledge as quick as possible, which we consider is more important for reinforcement learning at early stage. In this paper, we verified our hypothesis through experiments, and a Two-Stage Population Based Training (TS-PBT) method is proposed, which is a more efficient population based training method for deep reinforcement learning. Experiments show that at the same computational budget, our TS-PBT method makes the final performance of the model significantly better than the PBT method. TS-PBT achieved 40%, 310%, 2%, 53%, 30% and 38% performance improvement over PBT separately in six test environments.

Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning

STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularization

Two-stage Population Based Training Method for Deep Reinforcement Learning

Efficient Online Hyperparameter Adaptation for Deep Reinforcement Learning.

Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning

Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy

Towards Autonomous Reinforcement Learning: Automatic Setting of Hyper-parameters using Bayesian Optimization

Data Efficient Training for Reinforcement Learning with Adaptive Behavior Policy Sharing

Efficient Hyperparameter Optimization with Probability-based Resource Allocating on Deep Neural Networks

Efficient hyperparameters optimization through model-based reinforcement learning with experience exploiting and meta-learning

Multi-Objective Population Based Training

Hyperparameters in Reinforcement Learning and How To Tune Them

Hyp-RL : Hyperparameter Optimization by Reinforcement Learning

Hyperparameter Optimization for Multi-Objective Reinforcement Learning

Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloning

AutoRL Hyperparameter Landscapes

Hyperparameter Optimization for Driving Strategies Based on Reinforcement Learning

ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning

Deep Reinforcement Learning using Genetic Algorithm for Parameter Optimization

Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning

Hyper: Hyperparameter Robust Efficient Exploration in Reinforcement Learning