Abstract:In realistic sparse reward tasks, existing theoretical methods cannot be effectively applied due to the low sampling probability ofrewarded episodes. Profound research on methods based on intrinsic rewards has been conducted to address this issue, but exploration with sparse rewards remains a great challenge. This paper describes the loop enhancement effect in exploration processes with sparse rewards. After each fully trained iteration, the execution probability of ineffective actions is higher than thatof other suboptimal actions, which violates biological habitual behavior principles and is not conducive to effective training. This paper proposes corresponding theorems of relieving the loop enhancement effect in the exploration process with sparse rewards and a heuristic exploration method based on action effectiveness constraints (AEC), which improves policy training efficiency by relieving the loop enhancement effect. Inspired by the fact that animals form habitual behaviors and goal-directed behaviors through the dorsolateral striatum and dorsomedial striatum. The function of the dorsolateral striatum is simulated by an action effectiveness evaluation mechanism (A2EM), which aims to reduce the rate of ineffective samples and improve episode reward expectations. The function of the dorsomedial striatum is simulated by an agent policy network, which aims to achieve task goals. The iterative training of A2EM and the policy forms the AEC model structure. A2EM provides effective samples for the agent policy; the agent policy provides training constraints for A2EM. The experimental results show that A2EM can relieve the loop enhancement effect and has good interpretability and generalizability. AEC enables agents to effectively reduce the loop rate in samples, can collect more effective samples, and improve the efficiency of policy training. The performance of AEC demonstrates the effectiveness of a biological heuristic approach that simulates the function of the dorsal striatum. This approach can be used to improve the robustness of agent exploration with sparse rewards.

A Temporally Correlated Latent Exploration for Reinforcement Learning

Random Latent Exploration for Deep Reinforcement Learning

Self-Supervised Exploration via Temporal Inconsistency in Reinforcement Learning

Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments

Leveraging Transition Exploratory Bonus for Efficient Exploration in Hard-Transiting Reinforcement Learning Problems.

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration

Intrinsic Rewards for Exploration Without Harm From Observational Noise: A Simulation Study Based on the Free Energy Principle

DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards

Random curiosity-driven exploration in deep reinforcement learning

Self-Attention-Based Temporary Curiosity in Reinforcement Learning Exploration

Enhancing Curiosity-driven Reinforcement Learning Through Historical State Information for Long-term Exploration

Variable-Agnostic Causal Exploration for Reinforcement Learning

A Novel Heuristic Exploration Method Based on Action Effectiveness Constraints to Relieve Loop Enhancement Effect in Reinforcement Learning with Sparse Rewards

Temporal Difference Uncertainties as a Signal for Exploration

Intrinsic Motivation Exploration Via Self-Supervised Prediction in Reinforcement Learning

Contingency-Aware Exploration in Reinforcement Learning

Latent Exploration for Reinforcement Learning

Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

Efficient and Scalable Exploration Via Estimation-Error

Successor-Predecessor Intrinsic Exploration