An Adversarial Objective for Scalable Exploration

Bernadette Bucher,Karl Schmeckpeper,Nikolai Matni,Kostas Daniilidis

DOI: https://doi.org/10.48550/arXiv.2003.06082

2020-11-12

Abstract:Model-based curiosity combines active learning approaches to optimal sampling with the information gain based incentives for exploration presented in the curiosity literature. Existing model-based curiosity methods look to approximate prediction uncertainty with approaches which struggle to scale to many prediction-planning pipelines used in robotics tasks. We address these scalability issues with an adversarial curiosity method minimizing a score given by a discriminator network. This discriminator is optimized jointly with a prediction model and enables our active learning approach to sample sequences of observations and actions which result in predictions considered the least realistic by the discriminator. We demonstrate progressively increasing advantages as compute is restricted of our adversarial curiosity approach over leading model-based exploration strategies in simulated environments. We further demonstrate the ability of our adversarial curiosity method to scale to a robotic manipulation prediction-planning pipeline where we improve sample efficiency and prediction performance for a domain transfer problem.

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the computational limitation problem in existing model - based curiosity methods that are difficult to scale to many robotic tasks in the prediction - planning pipeline. Specifically, existing model - based curiosity methods usually rely on model ensembles to estimate prediction uncertainty. This method is difficult to implement in tasks that require a large amount of computational resources. For example, modern visual prediction methods may require the computing power of an entire GPU, and using model ensembles requires multiple GPUs, which is unrealistic for hardware - limited systems. To address this challenge, the authors propose an adversarial curiosity method. Actions are selected by minimizing the scores given by the discriminative network, and these actions lead to predictions that are judged to be the least realistic. This method can not only improve sample efficiency but also improve prediction performance, and is especially suitable for situations with limited computational resources. The paper verifies the effectiveness of this method through experiments in the simulation environment and the actual robotic operation prediction - planning pipeline, and shows its advantages over other leading model exploration strategies in computationally - constrained environments.

An Adversarial Objective for Scalable Exploration

Safe Sim-to-Real Robot Exploration with Constrained Bayesian Optimization

Model-Based Robot Learning Control with Uncertainty Directed Exploration

Curiosity-Driven Exploration via Latent Bayesian Surprise

CMBE: Curiosity-driven Model-Based Exploration for Multi-Agent Reinforcement Learning in Sparse Reward Settings

Adversarial Active Exploration for Inverse Dynamics Model Learning

CARE: Confidence-rich Autonomous Robot Exploration Using Bayesian Kernel Inference and Optimization

Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots

Meta-learning curiosity algorithms

Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments

Curiosity-driven Exploration by Self-supervised Prediction

Dynamic Neural Curiosity Enhances Learning Flexibility for Autonomous Goal Discovery

CIExplore: Curiosity and Influence-based Exploration in Multi-Agent Cooperative Scenarios with Sparse Rewards

Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning

Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

BYOL-Explore: Exploration by Bootstrapped Prediction

Curious Meta-Controller: Adaptive Alternation between Model-Based and Model-Free Control in Deep Reinforcement Learning

Active Exploration for Robotic Manipulation

Interesting Object, Curious Agent: Learning Task-Agnostic Exploration

Computational mechanisms of curiosity and goal-directed exploration

How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimation