Abstract:Reinforcement learning (RL) and Deep Reinforcement Learning (DRL), in particular, have the potential to disrupt and are already changing the way we interact with the world. One of the key indicators of their applicability is their ability to scale and work in real-world scenarios, that is in large-scale problems. This scale can be achieved via a combination of factors, the algorithm's ability to make use of large amounts of data and computational resources and the efficient exploration of the environment for viable solutions (i.e. policies). In this work, we investigate and motivate some theoretical foundations for deep reinforcement learning. We start with exact dynamic programming and work our way up to stochastic approximations and stochastic approximations for a model-free scenario, which forms the theoretical basis of modern reinforcement learning. We present an overview of this highly varied and rapidly changing field from the perspective of Approximate Dynamic Programming. We then focus our study on the short-comings with respect to exploration of the cornerstone approaches (i.e. DQN, DDQN, A2C) in deep reinforcement learning. On the theory side, our main contribution is the proposal of a novel Bayesian actor-critic algorithm. On the empirical side, we evaluate Bayesian exploration as well as actor-critic algorithms on standard benchmarks as well as state-of-the-art evaluation suites and show the benefits of both of these approaches over current state-of-the-art deep RL methods. We release all the implementations and provide a full python library that is easy to install and hopefully will serve the reinforcement learning community in a meaningful way, and provide a strong foundation for future work.

Bounded Exploration with World Model Uncertainty in Soft Actor-Critic Reinforcement Learning Algorithm

LiFE:Deep Exploration Via Linear-Feature Bonus in Continuous Control

Efficient and Scalable Exploration Via Estimation-Error

Exploration in Feature Space for Reinforcement Learning

Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning

DQN with model-based exploration: efficient learning on environments with sparse rewards

Leveraging Transition Exploratory Bonus for Efficient Exploration in Hard-Transiting Reinforcement Learning Problems.

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

Reward-Based Exploration: Adaptive Control for Deep Reinforcement Learning

BeBold: Exploration Beyond the Boundary of Explored Regions

MADE: Exploration via Maximizing Deviation from Explored Regions

Upper Confidence Bound Exploration with Fast Convergence

Learning Off-policy with Model-based Intrinsic Motivation For Active Online Exploration

Model-based Exploration Strategy to Accelerate Deterministic Strategy Algorithm Training

Dynamic Subgoal-based Exploration via Bayesian Optimization

Dynamic Bottleneck for Robust Self-Supervised Exploration

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Efficient Exploration in Resource-Restricted Reinforcement Learning

Never Give Up: Learning Directed Exploration Strategies