Abstract:The sparsity of extrinsic rewards presents a significant challenge for deep reinforcement learning (DRL). As an alternative, researchers have focused on intrinsic rewards to improve exploration efficiency. One of the most representative approaches is utilizing curiosity. However, the challenge of designing effective intrinsic rewards remains, as artificial curiosity differs significantly from human curiosity. In this article, we introduce a novel curiosity approach for DRL, named DyMeCu, which stands for Dynamic Memory-based Curiosity. Inspired by human curiosity and information theory, DyMeCu constructs a dynamic memory using the online learner following the bootstrap paradigm. Additionally, we design a two-learner architecture inspired by ensemble techniques to access curiosity better. The information gap between the two learners serves as the intrinsic reward for agents, and the state information is consistently consolidated into the dynamic memory. Compared with previous curiosity methods, DyMeCu can better mimic human curiosity by using a dynamic memory that can be dynamically grown based on a bootstrap paradigm with two learners. Large-scale empirical experiments on multiple benchmarks, including DeepMind Control Suite and Atari Suite, demonstrate that DyMeCu outperforms competitive curiosity-based methods with or without extrinsic rewards. In the Atari Suite, DyMeCu achieves a mean human-normalized score of 5.076 on a subset of 26 Atari games, achieving a 77.4% relative improvement over the best other baselines. In the DeepMind Control Suite, DyMeCu presents new state-of-the-art results across 11 tasks of all 12 when compared to curiosity-based methods and other pre-training strategies.

Curiosity-Driven Exploration via Latent Bayesian Surprise

Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments

Computational mechanisms of curiosity and goal-directed exploration

Curiosity-driven Exploration by Self-supervised Prediction

CMBE: Curiosity-driven Model-Based Exploration for Multi-Agent Reinforcement Learning in Sparse Reward Settings

How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimation

CIExplore: Curiosity and Influence-based Exploration in Multi-Agent Cooperative Scenarios with Sparse Rewards

Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

Intrinsic Rewards for Exploration Without Harm From Observational Noise: A Simulation Study Based on the Free Energy Principle

Meta-learning curiosity algorithms

Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learning

Self-Attention-Based Temporary Curiosity in Reinforcement Learning Exploration

Dynamic Memory-Based Curiosity: A Bootstrap Approach for Exploration in Reinforcement Learning

An Adversarial Objective for Scalable Exploration

Enhancing Curiosity-driven Reinforcement Learning Through Historical State Information for Long-term Exploration

Ask & Explore: Grounded Question Answering for Curiosity-Driven Exploration

Large-Scale Study of Curiosity-Driven Learning

Dynamic Memory-based Curiosity: A Bootstrap Approach for Exploration

Random curiosity-driven exploration in deep reinforcement learning

BYOL-Explore: Exploration by Bootstrapped Prediction

Explore and Control with Adversarial Surprise