Abstract:Traditional exploration methods in RL require agents to perform random actions to find rewards. But these approaches struggle on sparse-reward domains like Montezuma's Revenge where the probability that any random action sequence leads to reward is extremely low. Recent algorithms have performed well on such tasks by encouraging agents to visit new states or perform new actions in relation to all prior training episodes (which we call across-training novelty). But such algorithms do not consider whether an agent exhibits intra-life novelty: doing something new within the current episode, regardless of whether those behaviors have been performed in previous episodes. We hypothesize that across-training novelty might discourage agents from revisiting initially non-rewarding states that could become important stepping stones later in training. We introduce Deep Curiosity Search (DeepCS), which encourages intra-life exploration by rewarding agents for visiting as many different states as possible within each episode, and show that DeepCS matches the performance of current state-of-the-art methods on Montezuma's Revenge. We further show that DeepCS improves exploration on Amidar, Freeway, Gravitar, and Tutankham (many of which are hard exploration games). Surprisingly, DeepCS doubles A2C performance on Seaquest, a game we would not have expected to benefit from intra-life exploration because the arena is small and already easily navigated by naive exploration techniques. In one run, DeepCS achieves a maximum training score of 80,000 points on Seaquest, higher than any methods other than Ape-X. The strong performance of DeepCS on these sparse- and dense-reward tasks suggests that encouraging intra-life novelty is an interesting, new approach for improving performance in Deep RL and motivates further research into hybridizing across-training and intra-life exploration methods.

Random curiosity-driven exploration in deep reinforcement learning

Enhancing Curiosity-driven Reinforcement Learning Through Historical State Information for Long-term Exploration

Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems

Random Latent Exploration for Deep Reinforcement Learning

Reward Space Noise for Exploration in Deep Reinforcement Learning

Curiosity-Driven Rainbow Agent Exploratory Learning

Large-Scale Study of Curiosity-Driven Learning

Intrinsic Rewards for Exploration Without Harm From Observational Noise: A Simulation Study Based on the Free Energy Principle

Dynamic Memory-Based Curiosity: A Bootstrap Approach for Exploration in Reinforcement Learning

DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards

Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards

Intrinsic Motivation Exploration Via Self-Supervised Prediction in Reinforcement Learning

Curiosity-driven Exploration by Self-supervised Prediction

End-to-End Autonomous Exploration with Deep Reinforcement Learning and Intrinsic Motivation

Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

DQN with model-based exploration: efficient learning on environments with sparse rewards

Dynamic Memory-based Curiosity: A Bootstrap Approach for Exploration

Efficient and Scalable Exploration Via Estimation-Error

Self-Supervised Exploration via Temporal Inconsistency in Reinforcement Learning

CIExplore: Curiosity and Influence-based Exploration in Multi-Agent Cooperative Scenarios with Sparse Rewards

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning