Abstract:Deep reinforcement learning (DRL) and deep multiagent reinforcement learning (MARL) have achieved significant success across a wide range of domains, including game artificial intelligence (AI), autonomous vehicles, and robotics. However, DRL and deep MARL agents are widely known to be sample inefficient that millions of interactions are usually needed even for relatively simple problem settings, thus preventing the wide application and deployment in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how efficiently exploring the environment and collecting informative experiences that could benefit policy learning toward the optimal ones. This problem becomes more challenging in complex environments with sparse rewards, noisy distractions, long horizons, and nonstationary co-learners. In this article, we conduct a comprehensive survey on existing exploration methods for both single-agent RL and multiagent RL. We start the survey by identifying several key challenges to efficient exploration. Then, we provide a systematic survey of existing approaches by classifying them into two major categories: uncertainty-oriented exploration and intrinsic motivation-oriented exploration. Beyond the above two main branches, we also include other notable exploration methods with different ideas and techniques. In addition to algorithmic analysis, we provide a comprehensive and unified empirical comparison of different exploration methods for DRL on a set of commonly used benchmarks. According to our algorithmic and empirical investigation, we finally summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.

Survey of Reinforcement Learning Based on Human Prior Knowledge

Hybrid Reinforcement Learning Based on Human Preference and Advice for Efficient Robot Skill Learning

Human-Centered Reinforcement Learning: A Survey

Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey

A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges

Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning

Pretraining in Deep Reinforcement Learning: A Survey

Prioritized Experience-Based Reinforcement Learning With Human Guidance for Autonomous Driving

A Survey on Population-Based Deep Reinforcement Learning

Path Planning in an Unknown Environment Based on Deep Reinforcement Learning with Prior Knowledge

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

A Survey of Deep Reinforcement Learning in Video Games

Hierarchical Reinforcement Learning: A Survey and Open Research Challenges

Neurosymbolic Reinforcement Learning and Planning: A Survey

Overview of Reinforcement Learning from Knowledge Expression and Handling

Adaptively Shaping Reinforcement Learning Agents Via Human Reward

A Survey of Reinforcement Learning Techniques: Strategies, Recent Development, and Future Directions

Reinforcement Learning Algorithms: A brief survey

Leveraging Human Guidance for Deep Reinforcement Learning Tasks

A Survey on Applications of Reinforcement Learning in Spatial Resource Allocation

Hierarchical Reinforcement Learning with Abductive Planning