Abstract:Deep reinforcement learning (DRL) and deep multiagent reinforcement learning (MARL) have achieved significant success across a wide range of domains, including game artificial intelligence (AI), autonomous vehicles, and robotics. However, DRL and deep MARL agents are widely known to be sample inefficient that millions of interactions are usually needed even for relatively simple problem settings, thus preventing the wide application and deployment in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how efficiently exploring the environment and collecting informative experiences that could benefit policy learning toward the optimal ones. This problem becomes more challenging in complex environments with sparse rewards, noisy distractions, long horizons, and nonstationary co-learners. In this article, we conduct a comprehensive survey on existing exploration methods for both single-agent RL and multiagent RL. We start the survey by identifying several key challenges to efficient exploration. Then, we provide a systematic survey of existing approaches by classifying them into two major categories: uncertainty-oriented exploration and intrinsic motivation-oriented exploration. Beyond the above two main branches, we also include other notable exploration methods with different ideas and techniques. In addition to algorithmic analysis, we provide a comprehensive and unified empirical comparison of different exploration methods for DRL on a set of commonly used benchmarks. According to our algorithmic and empirical investigation, we finally summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.

Multi-Agent Exploration Via Self-Learning and Social Learning

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

S2rl

The Multi-Agent System Based on Reinforcement Learning

Self-Motivated Multi-Agent Exploration

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

Learning Efficient Multi-Agent Cooperative Visual Exploration

Subspace-Aware Exploration for Sparse-Reward Multi-Agent Tasks.

Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

Influence-Based Multi-Agent Exploration

Two Heads Are Better Than One: A Simple Exploration Framework for Efficient Multi-Agent Reinforcement Learning.

Multi-Agent Exploration of an Unknown Sparse Landmark Complex via Deep Reinforcement Learning

Underwater Multi-agent Cooperative Formation Hunting Based on Deep Reinforcement Learning

End-to-End Autonomous Exploration with Deep Reinforcement Learning and Intrinsic Motivation

Multi-Robot Cooperative Socially-Aware Navigation Using Multi-Agent Reinforcement Learning

CMBE: Curiosity-driven Model-Based Exploration for Multi-Agent Reinforcement Learning in Sparse Reward Settings

MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structure

Emergent Social Learning via Multi-agent Reinforcement Learning

LIGS: Learnable Intrinsic-Reward Generation Selection for Multi-Agent Learning

WToE: Learning When to Explore in Multiagent Reinforcement Learning

Multi-agent Exploration with Sub-state Entropy Estimation