Abstract:Deep reinforcement learning (DRL) and deep multiagent reinforcement learning (MARL) have achieved significant success across a wide range of domains, including game artificial intelligence (AI), autonomous vehicles, and robotics. However, DRL and deep MARL agents are widely known to be sample inefficient that millions of interactions are usually needed even for relatively simple problem settings, thus preventing the wide application and deployment in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how efficiently exploring the environment and collecting informative experiences that could benefit policy learning toward the optimal ones. This problem becomes more challenging in complex environments with sparse rewards, noisy distractions, long horizons, and nonstationary co-learners. In this article, we conduct a comprehensive survey on existing exploration methods for both single-agent RL and multiagent RL. We start the survey by identifying several key challenges to efficient exploration. Then, we provide a systematic survey of existing approaches by classifying them into two major categories: uncertainty-oriented exploration and intrinsic motivation-oriented exploration. Beyond the above two main branches, we also include other notable exploration methods with different ideas and techniques. In addition to algorithmic analysis, we provide a comprehensive and unified empirical comparison of different exploration methods for DRL on a set of commonly used benchmarks. According to our algorithmic and empirical investigation, we finally summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.

Open Problems and Modern Solutions for Deep Reinforcement Learning

Deep Reinforcement Learning

Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes

Reward Mechanism Design for Deep Reinforcement Learning-Based Microgrid Energy Management

Modern Deep Reinforcement Learning Algorithms

A Survey of Deep Reinforcement Learning in Video Games

A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges

Deep Reinforcement Learning in Nonstationary Environments With Unknown Change Points

Bridging the gap between Markowitz planning and deep reinforcement learning

Security and Privacy Issues in Deep Reinforcement Learning: Threats and Countermeasures

Reinforcement learning for robot research: A comprehensive review and open issues

Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications

An Overview on Algorithms and Applications of Deep Reinforcement Learning

Deep Reinforcement Learning: A Brief Survey

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

Decision Theory-Guided Deep Reinforcement Learning for Fast Learning

Reinforcement Learning in Practice: Opportunities and Challenges

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Deep Reinforcement Learning on Autonomous Driving Policy With Auxiliary Critic Network

Deep Reinforcement Learning for Motion Control Algorithms in Robotics