Abstract:Deep reinforcement learning (DRL) and deep multiagent reinforcement learning (MARL) have achieved significant success across a wide range of domains, including game artificial intelligence (AI), autonomous vehicles, and robotics. However, DRL and deep MARL agents are widely known to be sample inefficient that millions of interactions are usually needed even for relatively simple problem settings, thus preventing the wide application and deployment in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how efficiently exploring the environment and collecting informative experiences that could benefit policy learning toward the optimal ones. This problem becomes more challenging in complex environments with sparse rewards, noisy distractions, long horizons, and nonstationary co-learners. In this article, we conduct a comprehensive survey on existing exploration methods for both single-agent RL and multiagent RL. We start the survey by identifying several key challenges to efficient exploration. Then, we provide a systematic survey of existing approaches by classifying them into two major categories: uncertainty-oriented exploration and intrinsic motivation-oriented exploration. Beyond the above two main branches, we also include other notable exploration methods with different ideas and techniques. In addition to algorithmic analysis, we provide a comprehensive and unified empirical comparison of different exploration methods for DRL on a set of commonly used benchmarks. According to our algorithmic and empirical investigation, we finally summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.

Action Mapping for Reinforcement Learning in Continuous Environments with Constraints

Learning to Generate All Feasible Actions

Action Mapping: A Reinforcement Learning Method for Constrained-Input Systems

Representation learning for continuous action spaces is beneficial for efficient policy learning

Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces

Environment ( CGRA ) Agent ( Mapping algorithm ) Action : PE Interchange Reward :

Data-Flow Graph Mapping Optimization for CGRA with Deep Reinforcement Learning

Reinforcement learning for mapping instructions to actions

Model-Based Reinforcement Learning In Continuous Environments Using Real-Time Constrained Optimization

Continuous-action reinforcement learning with fast policy search and adaptive basis function selection

Active Exploration Deep Reinforcement Learning for Continuous Action Space with Forward Prediction

End-to-End Autonomous Exploration for Mobile Robots in Unknown Environments through Deep Reinforcement Learning

Continuous Control with Coarse-to-fine Reinforcement Learning

Comparing Action Aggregation Strategies in Deep Reinforcement Learning with Continuous Action

Continuous control with deep reinforcement learning

Hierarchical Reinforcement Learning for Kinematic Control Tasks with Parameterized Action Spaces

An Efficient Deep Reinforcement Learning Algorithm for Mapless Navigation with Gap-Guided Switching Strategy

Using Deep Reinforcement Learning for the Continuous Control of Robotic Arms

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

Overcoming Exploration: Deep Reinforcement Learning for Continuous Control in Cluttered Environments from Temporal Logic Specifications