Abstract:In the last few years, the research activity around reinforcement learning tasks formulated over environments with sparse rewards has been especially notable. Among the numerous approaches proposed to deal with these hard exploration problems, intrinsic motivation mechanisms are arguably among the most studied alternatives to date. Advances reported in this area over time have tackled the exploration issue by proposing new algorithmic ideas to generate alternative mechanisms to measure the novelty. However, most efforts in this direction have overlooked the influence of different design choices and parameter settings that have also been introduced to improve the effect of the generated intrinsic bonus, forgetting the application of those choices to other intrinsic motivation techniques that may also benefit of them. Furthermore, some of those intrinsic methods are applied with different base reinforcement algorithms (e.g. PPO, IMPALA) and neural network architectures, being hard to fairly compare the provided results and the actual progress provided by each solution. The goal of this work is to stress on this crucial matter in reinforcement learning over hard exploration environments, exposing the variability and susceptibility of avant-garde intrinsic motivation techniques to diverse design factors. Ultimately, our experiments herein reported underscore the importance of a careful selection of these design aspects coupled with the exploration requirements of the environment and the task in question under the same setup, so that fair comparisons can be guaranteed.

Intrinsic Motivation Exploration Via Self-Supervised Prediction in Reinforcement Learning

Self-Supervised Exploration via Temporal Inconsistency in Reinforcement Learning

Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration

Learning Off-policy with Model-based Intrinsic Motivation For Active Online Exploration

End-to-End Autonomous Exploration with Deep Reinforcement Learning and Intrinsic Motivation

DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards

Random curiosity-driven exploration in deep reinforcement learning

Curiosity-driven Exploration by Self-supervised Prediction

SEMI: Self-supervised Exploration Via Multisensory Incongruity

Self-Attention-Based Temporary Curiosity in Reinforcement Learning Exploration

An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environments

EMI: Exploration with Mutual Information

Intrinsically Motivated Self-supervised Learning in Reinforcement Learning

Spatial Consciousness Model of Intrinsic Reward in Partially Observable Environments

Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning

Intrinsically Guided Exploration in Meta Reinforcement Learning

Intrinsically-Motivated Reinforcement Learning: A Brief Introduction

Show me the Way: Intrinsic Motivation from Demonstrations

Self-Attention Based Temporal Intrinsic Reward for Reinforcement Learning

Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments

Deep intrinsically motivated exploration in continuous control