Abstract:In realistic sparse reward tasks, existing theoretical methods cannot be effectively applied due to the low sampling probability ofrewarded episodes. Profound research on methods based on intrinsic rewards has been conducted to address this issue, but exploration with sparse rewards remains a great challenge. This paper describes the loop enhancement effect in exploration processes with sparse rewards. After each fully trained iteration, the execution probability of ineffective actions is higher than thatof other suboptimal actions, which violates biological habitual behavior principles and is not conducive to effective training. This paper proposes corresponding theorems of relieving the loop enhancement effect in the exploration process with sparse rewards and a heuristic exploration method based on action effectiveness constraints (AEC), which improves policy training efficiency by relieving the loop enhancement effect. Inspired by the fact that animals form habitual behaviors and goal-directed behaviors through the dorsolateral striatum and dorsomedial striatum. The function of the dorsolateral striatum is simulated by an action effectiveness evaluation mechanism (A2EM), which aims to reduce the rate of ineffective samples and improve episode reward expectations. The function of the dorsomedial striatum is simulated by an agent policy network, which aims to achieve task goals. The iterative training of A2EM and the policy forms the AEC model structure. A2EM provides effective samples for the agent policy; the agent policy provides training constraints for A2EM. The experimental results show that A2EM can relieve the loop enhancement effect and has good interpretability and generalizability. AEC enables agents to effectively reduce the loop rate in samples, can collect more effective samples, and improve the efficiency of policy training. The performance of AEC demonstrates the effectiveness of a biological heuristic approach that simulates the function of the dorsal striatum. This approach can be used to improve the robustness of agent exploration with sparse rewards.

Optimized Feature Extraction for Sample Efficient Deep Reinforcement Learning

A Novel Adaptive Sampling Strategy for Deep Reinforcement Learning.

Random curiosity-driven exploration in deep reinforcement learning

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

Efficient and Scalable Exploration Via Estimation-Error

Efficient Novelty Search Through Deep Reinforcement Learning

LiFE:Deep Exploration Via Linear-Feature Bonus in Continuous Control

DQN with model-based exploration: efficient learning on environments with sparse rewards

Automated Feature Selection: A Reinforcement Learning Perspective

Reward Space Noise for Exploration in Deep Reinforcement Learning

Never Give Up: Learning Directed Exploration Strategies

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Sampling Efficient Deep Reinforcement Learning through Preference-Guided Stochastic Exploration

A Scalable Derivative-free Exploration Approach for Reinforcement Learning

A Novel Heuristic Exploration Method Based on Action Effectiveness Constraints to Relieve Loop Enhancement Effect in Reinforcement Learning with Sparse Rewards

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

A Data-Efficient Training Method for Deep Reinforcement Learning

Sample Efficient Deep Reinforcement Learning via Local Planning

Random Latent Exploration for Deep Reinforcement Learning

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

NAEM: Noisy Attention Exploration Module for Deep Reinforcement Learning