Abstract:In realistic sparse reward tasks, existing theoretical methods cannot be effectively applied due to the low sampling probability ofrewarded episodes. Profound research on methods based on intrinsic rewards has been conducted to address this issue, but exploration with sparse rewards remains a great challenge. This paper describes the loop enhancement effect in exploration processes with sparse rewards. After each fully trained iteration, the execution probability of ineffective actions is higher than thatof other suboptimal actions, which violates biological habitual behavior principles and is not conducive to effective training. This paper proposes corresponding theorems of relieving the loop enhancement effect in the exploration process with sparse rewards and a heuristic exploration method based on action effectiveness constraints (AEC), which improves policy training efficiency by relieving the loop enhancement effect. Inspired by the fact that animals form habitual behaviors and goal-directed behaviors through the dorsolateral striatum and dorsomedial striatum. The function of the dorsolateral striatum is simulated by an action effectiveness evaluation mechanism (A2EM), which aims to reduce the rate of ineffective samples and improve episode reward expectations. The function of the dorsomedial striatum is simulated by an agent policy network, which aims to achieve task goals. The iterative training of A2EM and the policy forms the AEC model structure. A2EM provides effective samples for the agent policy; the agent policy provides training constraints for A2EM. The experimental results show that A2EM can relieve the loop enhancement effect and has good interpretability and generalizability. AEC enables agents to effectively reduce the loop rate in samples, can collect more effective samples, and improve the efficiency of policy training. The performance of AEC demonstrates the effectiveness of a biological heuristic approach that simulates the function of the dorsal striatum. This approach can be used to improve the robustness of agent exploration with sparse rewards.

Policy Augmentation: An Exploration Strategy for Faster Convergence of Deep Reinforcement Learning Algorithms

Generalize Robot Learning from Demonstration to Variant Scenarios with Evolutionary Policy Gradient

Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning

Multi-Agent Deep Deterministic Policy Gradient Algorithm Based on Classification Experience Replay

A Scalable Derivative-free Exploration Approach for Reinforcement Learning

PP-PG: Combining Parameter Perturbation with Policy Gradient Methods for Effective and Efficient Explorations in Deep Reinforcement Learning

Improved Exploration through Latent Trajectory Optimization in Deep Deterministic Policy Gradient

Adversarial Policy Optimization in Deep Reinforcement Learning

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

Diverse Exploration for Fast and Safe Policy Improvement

A Novel Policy Based on Action Confidence Limit to Improve Exploration Efficiency in Reinforcement Learning

Towards More Sample Efficiency in Reinforcement Learning with Data Augmentation.

Curious Explorer: a Provable Exploration Strategy in Policy Learning

Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning

Policy Gradient from Demonstration and Curiosity

Behind the Myth of Exploration in Policy Gradients

Invariant Transform Experience Replay: Data Augmentation for Deep Reinforcement Learning.

Data Efficient Training for Reinforcement Learning with Adaptive Behavior Policy Sharing

A Novel Heuristic Exploration Method Based on Action Effectiveness Constraints to Relieve Loop Enhancement Effect in Reinforcement Learning with Sparse Rewards

Automatic Data Augmentation for Generalization in Deep Reinforcement Learning

Never Give Up: Learning Directed Exploration Strategies