Abstract:Deep reinforcement learning (DRL) and deep multiagent reinforcement learning (MARL) have achieved significant success across a wide range of domains, including game artificial intelligence (AI), autonomous vehicles, and robotics. However, DRL and deep MARL agents are widely known to be sample inefficient that millions of interactions are usually needed even for relatively simple problem settings, thus preventing the wide application and deployment in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how efficiently exploring the environment and collecting informative experiences that could benefit policy learning toward the optimal ones. This problem becomes more challenging in complex environments with sparse rewards, noisy distractions, long horizons, and nonstationary co-learners. In this article, we conduct a comprehensive survey on existing exploration methods for both single-agent RL and multiagent RL. We start the survey by identifying several key challenges to efficient exploration. Then, we provide a systematic survey of existing approaches by classifying them into two major categories: uncertainty-oriented exploration and intrinsic motivation-oriented exploration. Beyond the above two main branches, we also include other notable exploration methods with different ideas and techniques. In addition to algorithmic analysis, we provide a comprehensive and unified empirical comparison of different exploration methods for DRL on a set of commonly used benchmarks. According to our algorithmic and empirical investigation, we finally summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.

Algorithms in Multi-Agent Systems: A Holistic Perspective from Reinforcement Learning and Game Theory

A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications

Multi-agent Reinforcement Learning: A Comprehensive Survey

An Overview of Multi-Agent Reinforcement Learning from Game Theoretical Perspective

Deep multiagent reinforcement learning: challenges and directions

Multi-Agent Reinforcement Learning: A Report on Challenges and Approaches

A review of research on reinforcement learning algorithms for multi-agents

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey

Multi-Agent Deep Reinforcement Learning for Multi-Robot Applications: A Survey

Deep reinforcement learning algorithm based on multi-agent parallelism and its application in game environment

Multi-Agent Evolutionary Reinforcement Learning Based on Cooperative Games

A Survey of Deep Reinforcement Learning in Video Games

Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques

A Two-Layered Multi-Agent Reinforcement Learning Model and Algorithm

Reinforcement learning and adaptive/approximate dynamic programming: A survey from theory to applications in multi-agent systems

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

LMRL: a Multi-Agent Reinforcement Learning Model and Algorithm

A Survey on Self-play Methods in Reinforcement Learning

A Survey and Critique of Multiagent Deep Reinforcement Learning

Overview on reinforcement learning of multi-agent game