Abstract:With shaped reward functions, reinforcement learning (RL) has recently been successfully applied to several robot control tasks. However, designing a task-relevant and well-performing reward function takes time and effort. Still, if RL can train an agent to complete a task in a sparse reward environment, it is an effective way to address the difficulty of reward function design, but it is still a significant challenge. To address this issue, the pioneering hindsight experience replay (HER) method dramatically enhances the probability of acquiring skills in sparse reward environments by transforming unsuccessful experiences into helpful training samples. However, HER still requires a lengthy training period. In this article, we propose a new technique based on HER termed adaptive HER with goal-amended curiosity module (AHEGC) for further enhancing sample and exploration efficiency. Specifically, an adaptive adjustment strategy of hindsight experience (HE) sampling rate and reward weights is developed to enhance sample efficiency. Furthermore, we introduce a curiosity mechanism to encourage more efficient exploration of the environment and propose a goal-amended (GA) curiosity module as a solution to the problem of over-seeking novelty caused by the curiosity introduced. We conducted experiments on six demanding robot control tasks with binary rewards, including Fetch and Hand environments. The results show that the proposed method outperforms existing methods regarding learning ability and convergence speed.

Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning

MHER: Model-based Hindsight Experience Replay

Combining Hindsight with Goal-enhanced Prediction for Multi-goal Reinforcement Learning

MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewards

Addressing Hindsight Bias in Multigoal Reinforcement Learning

Quantile Regression Hindsight Experience Replay

Soft Hindsight Experience Replay

Guided Goal Generation for Hindsight Multi-Goal Reinforcement Learning

Learning and reusing primitive behaviours to improve Hindsight Experience Replay sample efficiency

AHEGC: Adaptive Hindsight Experience Replay with Goal-Amended Curiosity Module for Robot Control.

Clustering-based Failed goal Aware Hindsight Experience Replay

Hindsight Planner.

ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning

Deep Reinforcement Learning for Complex Manipulation Tasks with Sparse Feedback

Exploration via Hindsight Goal Generation

Generating Attentive Goals for Prioritized Hindsight Reinforcement Learning

Consistent Experience Replay in High-Dimensional Continuous Control with Decayed Hindsights

Unbiased Methods for Multi-Goal Reinforcement Learning

Relay Hindsight Experience Replay: Self-guided continual reinforcement learning for sequential object manipulation tasks with sparse rewards

Diversity-based Trajectory and Goal Selection with Hindsight Experience Replay

A Guided Evaluation Method for Robot Dynamic Manipulation