Abstract:Hierarchical reinforcement learning (HRL) is a promising method to extend traditional reinforcement learning to solve more complex tasks. HRL can solve the problems of long-term reward sparsity and credit assignment. However, the existing HRL methods are trained in specific environments and target tasks each time, resulting in low sample utilization. In addition, the low-level sub-policies of the agent will interfere with each other during the migration process, resulting in poor policy stability. Aiming at the issue above, this paper proposes an HRL method, Relabeling and Policy Distillation of Hierarchical Reinforcement Learning (R-PD-HRL), that integrates meta-learning, shared reward relabeling and policy distillation to accelerate the learning speed and improve the policy stability of the agent. In the training process, a reward relabeling module is introduced to act on the experience buffer. Different reward functions are used to relabel the interaction trajectory for the training of other tasks under the same task distribution. At the low-level, policy distillation technology is used to compress the sub-policies of the low-level, and the interference between the policies is reduced while ensuring the correctness of the original low-level sub-policies. Finally, according to different tasks, the high-level policy calls the low-level optimal policy to complete the decision. In both continuous and discrete state-action environments, experimental results show that compared with other methods, the improved sample utilization of this method greatly accelerates the learning speed, and the success rate is as high as 0.6.

Relabeling and policy distillation of hierarchical reinforcement learning

Boosting Reinforcement Learning via Hierarchical Game Playing With State Relay

Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

Data-Efficient Hierarchical Reinforcement Learning for Robotic Assembly Control Applications

Temporal-adaptive Hierarchical Reinforcement Learning

Efficient Hierarchical Policy Network with Fuzzy Rules

Algorithms for Batch Hierarchical Reinforcement Learning

Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policies

HRL2E: Hierarchical Reinforcement Learning with Low-level Ensemble

$\epsilon$-Invariant Hierarchical Reinforcement Learning for Building Generalizable Policy

HLifeRL: A Hierarchical Lifelong Reinforcement Learning Framework

Meta-Learning Integration in Hierarchical Reinforcement Learning for Advanced Task Complexity

Hierarchical Reinforcement Learning from Demonstration via Reachability-Based Reward Shaping

Reinforcement Learning via Auxiliary Task Distillation

Diversity-Driven Extensible Hierarchical Reinforcement Learning.

Integrating human learning and reinforcement learning: A novel approach to agent training

Deep Reinforcement Learning from Hierarchical Preference Design

RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning

Privileged Knowledge Distillation for Sim-to-Real Policy Generalization

Goal-Conditioned Hierarchical Reinforcement Learning with High-Level Model Approximation.

On Credit Assignment in Hierarchical Reinforcement Learning