Abstract:The sparsity of extrinsic rewards presents a significant challenge for deep reinforcement learning (DRL). As an alternative, researchers have focused on intrinsic rewards to improve exploration efficiency. One of the most representative approaches is utilizing curiosity. However, the challenge of designing effective intrinsic rewards remains, as artificial curiosity differs significantly from human curiosity. In this article, we introduce a novel curiosity approach for DRL, named DyMeCu, which stands for Dynamic Memory-based Curiosity. Inspired by human curiosity and information theory, DyMeCu constructs a dynamic memory using the online learner following the bootstrap paradigm. Additionally, we design a two-learner architecture inspired by ensemble techniques to access curiosity better. The information gap between the two learners serves as the intrinsic reward for agents, and the state information is consistently consolidated into the dynamic memory. Compared with previous curiosity methods, DyMeCu can better mimic human curiosity by using a dynamic memory that can be dynamically grown based on a bootstrap paradigm with two learners. Large-scale empirical experiments on multiple benchmarks, including DeepMind Control Suite and Atari Suite, demonstrate that DyMeCu outperforms competitive curiosity-based methods with or without extrinsic rewards. In the Atari Suite, DyMeCu achieves a mean human-normalized score of 5.076 on a subset of 26 Atari games, achieving a 77.4% relative improvement over the best other baselines. In the DeepMind Control Suite, DyMeCu presents new state-of-the-art results across 11 tasks of all 12 when compared to curiosity-based methods and other pre-training strategies.

Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning

CIExplore: Curiosity and Influence-based Exploration in Multi-Agent Cooperative Scenarios with Sparse Rewards

CMBE: Curiosity-driven Model-Based Exploration for Multi-Agent Reinforcement Learning in Sparse Reward Settings

Curiosity-driven Exploration by Self-supervised Prediction

Never Explore Repeatedly in Multi-Agent Reinforcement Learning

Random curiosity-driven exploration in deep reinforcement learning

Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

Enhancing Curiosity-driven Reinforcement Learning Through Historical State Information for Long-term Exploration

Multi-agent Collaborative Target Search Based on Curiosity Intrinsic Motivation

Self-Supervised Exploration via Temporal Inconsistency in Reinforcement Learning

Dealing with Sparse Rewards in Reinforcement Learning

Self-Attention-Based Temporary Curiosity in Reinforcement Learning Exploration

Large-Scale Study of Curiosity-Driven Learning

Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments

Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

A unified strategy for implementing curiosity and empowerment driven reinforcement learning

Never Revisit: Continuous Exploration in Multi-Agent Reinforcement Learning

Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration

Dynamic Memory-Based Curiosity: A Bootstrap Approach for Exploration in Reinforcement Learning

Intrinsic Motivation Exploration Via Self-Supervised Prediction in Reinforcement Learning

Dynamic Memory-based Curiosity: A Bootstrap Approach for Exploration