Abstract:In reinforcement learning (RL), agents learn policies from spatiotemporal data generated by interaction with the environment. However, spatiotemporal data containing reward signals are often sparse, the agent only gets a reward signal when it reaches the goal state. This kind of problem is technically challenging since rewards are the basis of policy learning. Recently, the prior knowledge of the task structure has been used to solve such problems such as hierarchical learning and goal-conditioned learning. In this paper, we consider a common task structure called Hard-Transiting. In a Hard-Transiting task, the difficulty of moving forward of the agent increases as it approaches the goal state. We formalize the Hard-Transiting RL problem with sparse rewards in which the transition probability decreases as the agent approaches the goal state. Inspired by reward bonus in interactive spatiotemporal data, we propose two novel algorithms with efficient exploration to solve such problems. For tabular setting, we adopt Transition Exploratory Bonus (TEB) to encourage exploration in Hard-Transiting problems and propose Model-Based Interval Estimation-TEB (MBIE-TEB) in which TEB is considered in the value iteration phase of the conventional MBIE algorithm. And under the performance metric of sample complexity, we give a theoretical proof of the upper bound of MBIE-TEB. For non-tabular setting, we propose Deep Q-Network-TEB (DQN-TEB) in which TEB is used as the intrinsic motivation in DQN. We test the proposed algorithms on two numerical tasks and one large-scale task. And the experimental results demonstrate that with the transition exploratory bonus, the proposed algorithms outperform the compared algorithms.

A Study of Count-Based Exploration and Bonus for Reinforcement Learning

A Dynamic Adjusting Reward Function Method for Deep Reinforcement Learning with Adjustable Parameters

Count-Based Exploration Via Embedded State Space for Deep Reinforcement Learning

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

Reward-Based Exploration: Adaptive Control for Deep Reinforcement Learning

Dynamic Bottleneck for Robust Self-Supervised Exploration

Exploration in Feature Space for Reinforcement Learning

Unifying Count-Based Exploration and Intrinsic Motivation

CMBE: Curiosity-driven Model-Based Exploration for Multi-Agent Reinforcement Learning in Sparse Reward Settings

BeBold: Exploration Beyond the Boundary of Explored Regions

Careful at Estimation and Bold at Exploration

Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning

Dynamic Subgoal-based Exploration via Bayesian Optimization

Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment

Leveraging Transition Exploratory Bonus for Efficient Exploration in Hard-Transiting Reinforcement Learning Problems.

Subspace-Aware Exploration for Sparse-Reward Multi-Agent Tasks.

MADE: Exploration via Maximizing Deviation from Explored Regions

VDSC: Enhancing Exploration Timing with Value Discrepancy and State Counts

Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning

Efficient and Scalable Exploration Via Estimation-Error

Backtracking Exploration for Reinforcement Learning